NOTE: This article is now depreciated. The folks who maintain the host list have stopped maintaining because of health reasons.
I surf the web an awful lot, probably slightly more than your average 13 year old geek. I notice that a lot of sites load rather slowly mostly because your waiting on content from outside the specific domain. For example if you go to a website like thechive.com (one of my favorites) you will notice it takes quite a long time loading the ads. It would be nice if you could block advertisements... oh you can?
Although I mentioned thechive.com I spend most of my time on the net looking for information, not entertainment. These ads really hinder my search speed!
So here is a quick way you can block all the ads. Not only will your surfing be faster but you will also save some bandwidth.
First off I would like to thank the fine folks at http://winhelp2002.mvps.org/ for doing all the leg work and collecting all the data necessary for this to work.
Now this is simple.
First, let’s make a copy of your current hosts file. Some people still use this!
Everything done below should be done as root, or with sudo access.
Here we will make a copy of your /etc/hosts file and save it in your home directory in a hidden file called .etchosts , but you can save it anywhere you like. Just be sure to change the script to make the location of your file.
cp /etc/hosts ~/.etchosts
Now we will make the shell script.
run:
vi /root/update_hosts.sh
Fill the file with the following
#!/bin/bash
cd /tmp
wget http://winhelp2002.mvps.org/hosts.txt
rm /etc/hosts
mv hosts.txt /etc/hosts
cat ~/.etchosts >> /etc/hosts
Now we have to make sure the script is executable:
chmod +x update_hosts.sh
So just a little explanation of what the script does, it’s simple.
1) cd /tmp (Changes to a temporary working directory)
2) wget http://winhelp2002.mvps.org/hosts.txt (Gets the hosts.txt file from mvps.org and saves it.
3) rm /etc/hosts (Deletes the current /etc/hosts file)
4) mv hosts.txt /etc/hosts (Moves the new downloaded file to the /etc/hosts file)
5) cat ~/.etchosts >> /etc/hosts (Moves your old host entries back into the new file)
Now all you have to do is run the script and your advertisements will disappear from your browser.
You can use a cron job to update the hosts file automatically every night. The good folks at mvps.org update the file fairly regularly, not everyday, but a couple times a month.
Let’s add it to roots cron.
run:
crontab -e
Then enter the following in the file.
59 23 * * * /root/update_hosts.sh
Remember, if you need to add something to your hosts file you now want to add it to ~/.etchosts and then run the script to update /etc/hosts.
REFERENCE: http://winhelp2002.mvps.org/hosts.htm
Right up to running crontab, none of that should be done as root. You can do the earlier bit as a normal user. I'd recommend nano or jed rather than vi, much easier to learn.
sudo su
cat /etc/hosts >/etc/hosts.orig
cat >/usr/local/bin/update_hosts.sh </etc/hosts
cat /etc/hosts.orig >>/etc/hosts
EOF
chmod +x /usr/local/bin/update_hosts.sh
crontab -e
59 23 * * * /usr/local/bin/update_hosts.sh
exit
Just a note of thanks for creating such clear, well written, helpful information. I've successfully implemented your instructions on my Centos 6.2 distro. Excellent results!
Sorry, how does this vi thing work.
Having trouble here
vi is a command line text editor. You can also use gedit or any text editor you like. If you want to learn vi, read this:
http://acms.ucsd.edu/info/vi_tutorial.html
KWrite is a popular text Editor, on KDE, and many Systems.
You could also use ABP on firefox or the same on Chromium.
Regards.
Encouraging users to eschew vi for lame editors like nano or jed (jed?) is a critical mistake. Failing to learn ex/vi may someday leave you stranded with an unsolvable problem if your system becomes hosed and editing a file is the only solution, but that system only has ex/vi installed. vi is the preeminent editor on ALL unix systems. But then, you really had nothing USEFUL to add to the mans script, did you?
You can do just about anything as root, as long as you're not doing anything stupid. I'd argue that's perhaps more risky to have a "sudo" than do everything (specially withoug password) than to use just "su". But I don't know for sure, I may be missing something. To me it seems that Ubuntu-style omnipotent sudo is more about convenience than safety. With some distros' original sudo config you wont even be able to do barely anything with just sudo by default, su is required, at very least to define what each user can sudo.
I don't think I've ever seen one that has not nano as default. Whenever that's the case/there's no nano, then, well, one who does not know "vi" may have to use some other PC and learn the basics/make some cheatsheet in a piece of paper and then go back to the troubled machine.
It's perhaps not the ideal scenario, but it's not like encouraging people to walk on African jungles without knowing how to shoot a rifle. Most of the time you can go by perfectly fine without ever learning vi, somewhat like you don't need to be a mechanic to have a car, and even to know how to do some basic maintenance.
You shouldn't remove your own/default /etc/hosts
It should suffice to:
wget http://winhelp2002.mvps.org/hosts.txt
cat hotst.txt >> /etc/hosts
rm hosts.txt
">>" will only add to the extant file
Then you edit your hosts file and remove the redundant lines, such as
127.0.0.1 localhost
I don't know if it matters so much that there are those redundancies though.
By not deleting your own hosts you won't lose some lines that may be setting the domain and whatnot. You might have lines somewhat like:
127.0.1.1 debian-joejoejoe.WORKGROUP debian-joejoejoe
Whatever it does, I'm not an expert on that.
Those would have been erased in the other method, and things like samba or network folders might not work I guess.
But if you're going to do that anyway, you only need to
rm /etc/hosts/
wget -O /etc/hosts
no need to move and rename everything all around the place.
If you want to make a script that does that periodically, you may want to consider to have a "hosts.header" (you name it) file with your current/default hosts file, and then have a script (called by cron) that will do something along the lines of
wget -O /tmp/newhosts (or /dev/shm, wherever works)
cat /etc/hosts.header > /etc/hosts (note the single, not double, ">", it erases the old file, so no "rm" is needed)
tail -n +26 /tmp/newhosts >> /etc/hosts (now it's a double, ">>", as it's only adding new stuff to the end of the file)
"tail -n +26" will trim cut out the 26 first lines of the downloaded hosts file, wich will be added to your own custom "header" via ">>".
Always backup the relevant files before testing new things.
but basically, you don't have anything useful to add to the mans script, do you?
After following the manual, I see a ^M after every entry in /etc/hosts, like,
127.0.0.1 ads.doubleclick.com^M
Just wanted to know if its normal. Its working perfectly though.
@Anonymous - usually this occurs when the file was created on Windows. You can remove the ^M by using dos2unix.
http://linux.about.com/od/commands/l/blcmdl1_dos2uni.htm
Thank you for the instructions. Worked well on my Linuxmint 16.
This is my solution, based on what I've seen in this thread. These instruction work in Ubuntu from a command prompt.
1) Create a copy of your original hosts file to use as a header:
sudo cp /etc/hosts /etc/hosts.header
2) Create the script to update the hosts file:
nano update_hosts.sh
3) Enter the following text in nano:
#!/bin/bash
#Change to a temprary working directory
cd /tmp
#Download the current hosts.txt file
wget -q http://winhelp2002.mvps.org/hosts.txt
#Overwrite the hosts file with the hosts header
cat /etc/hosts.header > /etc/hosts
#Append the hosts.txt data to the hosts file
tail -n +26 /tmp/hosts.txt >> /etc/hosts
#Delete hosts.txt
rm /tmp/hosts.txt
#Exit the script
exit
4) Save the file with CTRL-X in nano
5) Change ownership of the file to root and make the file executable:
sudo chown root:root /path/to/update_hosts.sh
sudo chmod 700 /path/to/update_hosts.sh
6) Ad an exception for update_hosts.sh to sudoers
sudo visudo
7 Below the line that reads '%sudo ALL=(ALL:ALL) ALL', enter the following:
user ALL=(ALL) NOPASSWD: /path/to/update_hosts.sh
*Replace 'user' with your username
8) You should now be able to execute the script without a sudo password by entering the following:
sudo ./path/to/update_hosts.sh
Just wanna say thx. Using your solution since years. 🙂
I decided to write a little script to update the hosts file with new changes and preserve any custom hosts entries:
#!/bin/bash
if [ "$1" == "--update" ] || [ "$1" == "-u" ] || [ "$1" == "-U" ]
then
echo ""
echo "Updating hosts file with changes from /etc/hosts.orig"
echo ""
echo "Backing up original hosts file"
cp /etc/hosts /etc/hosts.bak
Done="False"
while [ "$Done" == "False" ]
do
echo ""
echo "#Don't edit this file directly instead edit /etc/hosts.orig and run hostsupdate --update"
echo ""
cat /etc/hosts.orig
cat /etc/hosts.txt
Done="True"
done > /tmp/finalHosts.txt
echo "Deleting hosts"
rm -rf /etc/hosts
echo "Hosts deleted"
echo ""
echo "Moving new hosts file"
mv /tmp/finalHosts.txt /etc/hosts
echo "Done!"
echo ""
echo "Your hosts file has been successfully updated"
echo ""
echo "If things go wrong please restore your original hosts file from /etc/hosts.bak"
echo ""
exit
fi
wget http://winhelp2002.mvps.org/hosts.txt -P /tmp -N
CurrentHosts=$(/usr/bin/md5sum --tag /etc/hosts.txt)
CurrentHosts="${CurrentHosts##* }"
echo "Current Hosts File:"
echo "$CurrentHosts"
echo ""
echo "New Hosts File"
NewHosts=$(/usr/bin/md5sum --tag /tmp/hosts.txt)
NewHosts="${NewHosts##* }"
echo "$NewHosts"
if [ "$CurrentHosts" != "$NewHosts" ]
then
echo ""
echo "DIFFERENT"
rm -rf /etc/hosts.txt
cp /tmp/hosts.txt /etc/hosts.txt
echo "Backing up original hosts file"
cp /etc/hosts /etc/hosts.bak
echo ""
echo "Hosts file backed up at /etc/hosts.bak"
Done="False"
while [ "$Done" == "False" ]
do
echo ""
echo "#Don't edit this file directly instead edit /etc/hosts.orig and run hostsupdate --update"
echo ""
cat /etc/hosts.orig
cat /tmp/hosts.txt
Done="True"
done > /tmp/finalHosts.txt
echo "Deleting hosts"
rm -rf /etc/hosts
echo "Hosts deleted"
echo ""
echo "Moving new hosts file"
mv /tmp/finalHosts.txt /etc/hosts
echo "Done!"
echo ""
echo "Your hosts file has been successfully updated"
echo ""
echo "If things go wrong please restore your original hosts file from /etc/hosts.bak"
echo ""
else
echo ""
echo "SAME"
rm -rf /tmp/hosts.txt
echo ""
fi
It basically stores the latest adblocking hosts file at /etc/hosts.txt and downloads the latest file to /tmp. The md5sum is checked and if they are the same nothing happens. If they aren't the same then any hosts entries in /etc/hosts.orig are merged with the latest adblocking hosts files allowing for custom entries to be stored in /etc/hosts.orig and be saved everytime there is an update to the adblocking hosts file.
Wow no verification of certificate and that winhelp2002.mvps.org resolves to the correct server?
wget http://winhelp2002.mvps.org/hosts.txt
should be
wget https://winhelp2002.mvps.org/hosts.txt
but guess the site doesn't have it on SSL
An that to dl the file that overwrites your DNS server...
Don't be surprised if your bank site starts having certificate errors.
The above comments makes half sense. You are correct, it would be much safer with an SSL connection, but it is not offered. Other than that I am not sure what your talking about. There is no overwriting a DNS server, this is just adding static entries to your hosts file. Also it has no effect on banks or certificates.
Update to - Sean DavisSeptember 14, 2013 at 4:23 PM post
Sean im new to Linux and I made a change see below:
FROM - sudo ./path/to/update_hosts.sh
TO - sudo /path/to/update_hosts.sh
I have just removed the full stop after sudo and it works, such a great feeling.
Russ
"I notice that a lot of sites load rather slowly mostly because your waiting on content from outside the specific domain."
Typo: needs to be you're, instead of your.
Another Anonymous here:
You clearly don't understand what the person above is talking about. It is trivial to redirect a user to a bogus banking site. This is also not just adding static entries as you put it but replacing the original hosts file. It is also potentially dangerous and a violation of any reasonable security policy to download and execute scripts from random sources as root.
I agree again... We are not replacing the original hosts file, but appending to it. Either way you are correct, there is a chance that someone could feed a nasty file into mvps.org site and redirect your traffic. I also agree, that is is potentially dangerous to execute scripts downloaded from random sources as root. I would suggest people read any scripts, understand what each line is doing and use it only if they feel comfortable.
How about:
#!/bin/bash
cd /tmp
wget http://winhelp2002.mvps.org/hosts.txt
cat /tmp/hosts.txt >> /etc/hosts
The hijacking can be easily netered by forcefully rewriting the IP at the beginning of each line to 0.0.0.0
Even better: refuse accepting the script if it contains anything else than 0.0.0.0
A potential hijacker trying to subvert a legitimate domain will only trigger a warning and prevent the update.
AnonymousJune 16, 2014 at 4:12 PM
"Wow no verification of certificate and that winhelp2002.mvps.org resolves to the correct server?"
He makes an excellent point.
Therefore this script needs to and can do more. It needs to read the first two characters of each line and accept only the following:
"# "
"#*"
"0."
"" (blank line)
"::" (In this case, it then needs to read the whole line and only accept "::1 localhost #[IPv6]")
"12" (In this case, it then needs to read the whole line and only accept "127.0.0.1 localhost")
ONLY if this validation passes should the file contents by inserted into your hosts file.
There are many ways to skin a cat, but I will leave it for someone else to write the MOST EFFICIENT method of doing this, considering there are over 15,000 lines to process!
BTW, I only use this hosts file on my RT-AC66U router, and then it blocks ads for all devices on the network, including Windows 7, but doesn't work for a WinXP test machine. I can't see anything different in IPCONFIG /ALL.
Any suggestions?
How about
cat hosts.txt | grep "0.0.0.0*" > hosts2.txt
If you haven't used Linux or Unix lont enough to have learned vi, then you really have no business administrating such a machine.
nano is sufficient for this task.
Are you sure your XP machine is using your router as it's DNS server, and not something else?
I am no programmer but for verifying that there are only 0. host adresses in the file, wouldn't it be easier to only check that there are no lines beginning by a digit between 1 and 9?
Something like this would work to test the integrity of the downloaded file:
# Download file
wget -q -O /tmp/hosts.new $remote_hosts_file
# Remove commented lines
egrep -v '^#' /tmp/hosts.new > /tmp/hosts.tmp
# Remove lines before hosts entries
sed -n -i '/0.0.0.0 /,$p' /tmp/hosts.tmp
# Perform malice check to see if anything points to anything other than 0.0.0.0
if [[ $(awk '{print $1}' /tmp/hosts.tmp | uniq) != '0.0.0.0' ]]; then
printf "File has been tampered with! Exiting!n"
exit 1
fi
Entire script I use at: https://gist.github.com/cptskyhawk/94dbc83aefdab631b37d
If you use the original verbatim script at the top, the "invisible" file, .etchosts, contains the text of the original or previous /etc/hosts file. This .etchosts file gets concatenated to the hosts file, effectively doubling the size of the /etc/hosts file every time the script is run successfully. This last line will use up disk storage very quickly and eventually die when it can no longer write the hosts file. This line should be removed from the script:
cat ~/.etchosts >> /etc/hosts
remove the above line.
^ well said Akulkis.
For those that like simplicity here is one line that can be added in root cron:
@montly wget -O- http://winhelp2002.mvps.org/hosts.txt > /etc/host
im getting this....
dietpi@DietPi:~$ sudo wget -O- http://winhelp2002.mvps.org/hosts.txt > /etc/hosts
-bash: /etc/hosts: Permission denied
dietpi@DietPi:~$
whats the problem?
wouldn't you need to rename the hosts.txt to plain "hosts"? I've never heard of it working with a file extension.....but I like this approach lol a one liner.....but fearing this wont work....just trying to avoid copy and pasting a hosts file across multiple headless raspberry pi's lol....
@Anonymous October 28, 2015 at 9:23 AM
That's not correct.
The .etchosts file contains your personal/manual entries you want in the /etc/hosts file. If you use the method/script by the OP, you should not edit the /etc/hosts file anymore but make all personal changes in the future in .etchosts. As every time your run the update script /etc/hosts will be created new:
1. you need the line you want to remove from the script (cat ~/.etchosts >> /etc/hosts) to get your personal/manual entries into /etc/hosts!
2. the size of the hosts file will always be: file size hosts.txt + file size .etchosts. It won't grow if there are now new entries in either of them, resp. if their size is not growing.
What distro are you using? Fedora does not recognize the ~/.etchosts file.
When does Fedora not recognize ~/.etchosts? What command(s) are you running? And from where (path)?
If you ONLY have created the script update_hosts.sh and run it, no distro will recognize ~/.etchosts as it does not exist.
You need to create it via
cp /etc/hosts ~/.etchosts
This will create a copy of your current /etc/hosts in YOUR home directory. So, if YOU run the script it should find/use it 🙂
Hope that helps 🙂
PS: If that command line (cp /etc/hosts ~/.etchosts) were part of the script, THEN Anonymous would be correct assuming the file gets bigger and bigger. But it's only a one time action to create .etchosts as a copy of the ORIGINAL hosts file.
I'm using this in Gentoo Linux. I put my original hosts file at:
/etc/conf.d/hosts_local
I had to merge dos2unix. I wanted the local info listed first. This is my current cron.monthly script:
#!/bin/bash
cd /tmp
mv /etc/hosts hosts.old
cp /etc/conf.d/hosts_local /etc/hosts
wget http://winhelp2002.mvps.org/hosts.txt
dos2unix -- hosts.txt
cat hosts.txt >> /etc/hosts
Personal user preference and choice don't matter, huh?
I thought that was the whole point of GNU/Linux.
But what do I know, I use Nano.
^Akulkis
That's just ignorant. The whole reason why alternatives like nano were developed in the first place, and added to the repos *everywhere*, was because it served the needs and wants of far more people than just those developers. Nano is a mature program that does what the users want, and what vi cannot.
The fact it's not core is either bigotry, or just simple numbers. In places where nano isn't available, you can get it.
For the record, I hate vi with a passion that cannot be described in the human language. Been around for decades and still haven't learned it, and administrate such machines daily. My lack of vi skills is wholly intended, as I availed myself of the repos and installed nano. Just as it was designed to do.
People that enjoy learning 32 trillion commands to enter into what *appears* to be an editor isn't evidence of superiority, but sadomasochism. Must be a fallout from the regex community 🙂
Perhaps we nano users are just the new Luddites, but I don't need to remember 32 trillion commands just to type normally. vi starts out and you can't just type into naturally right away? Insta-fail, and about the weirdest thing I've seen. Most of the time I don't complain, because there were others like me and we just use nano instead of saying that all vi users are idiots that have no business administrating Linux machines.
Lastly, for the record, it's a bad idea from a security perspective to update it nightly as suggested, in ROOT no less! It's not HTTPS, and it doesn't have a signature you could awk yourself and verify. Doing such a thing nightly just makes DNS poisoining your network and machine much easier.
If they want to do that, theire daily updates need to be signed.
The reason people are having trouble with ~/.etchosts is because cron is run as root, and for the root user, ~ is /root, not /home/yourusername.
Also, automatically appending files from your home directory to your hosts file is a big security concern, but I'm guessing most people don't care.
When does Fedora not recognize ~/.etchosts? What command(s) are you running? And from where (path)?
If you ONLY have created the script update_hosts.sh and run it, no distro will recognize ~/.etchosts as it does not exist.
You need to create it via
cp /etc/hosts ~/.etchosts
This will create a copy of your current /etc/hosts in YOUR home directory. So, if YOU run the script it should find/use it 🙂
Hope that helps 🙂
PS: If that command line (cp /etc/hosts ~/.etchosts) were part of the script, THEN Anonymous would be correct assuming the file gets bigger and bigger. But it's only a one time action to create .etchosts as a copy of the ORIGINAL hosts file.
Something that worked for me pretty well. Basically I took some of teh ideas here and threw them all together in a script.
To make this script work, it is important that the beginn of the "Ad Block Section" is marked with "### START ADBLOCK ###" and your hosts file begins with "# /etc/hosts". These two lines are the marker for your regular hosts content. everything below "### START ADBLOCK ###" is to be replaced with every update.
So basically your hosts file should look something like this:
# /etc/hosts: Local Host Database
# IPv4 and IPv6 localhost aliases
127.0.0.1 localhost
::1 localhost
### START ADBLOCK ###
And here is the script I came up with. It does nothing else than just taking teh part between the 2 markers, copies that into another file and merges the updates into the new file and writes all back replacing the hosts file.
#!/bin/bash
cd /root
cat /etc/hosts | sed -n "/# /etc/hosts/,/### START ADBLOCK ###/p" > etc_hosts
wget http://winhelp2002.mvps.org/hosts.txt
# Fix line endings
sed -i -e 's/r//g' hosts.txt
# Remove commented lines
egrep -v '^#' hosts.txt > hosts.tmp
# Remove lines before hosts entries
sed -n -i '/0.0.0.0 /,$p' hosts.tmp
# Perform malice check to see if anything points to anything other than 0.0.0.0
if [[ $(awk '{print $1}' hosts.tmp | uniq) != '0.0.0.0' ]]; then
printf "File has been tampered with! Exiting!n"
exit 1
fi
cp etc_hosts hosts_without_adblock
cat hosts.tmp >> etc_hosts
cat etc_hosts > /etc/hosts
rm hosts.txt
rm hosts.tmp
Hey buddy, you can remove redundant lines by running
awk '!x[$0]++' /tmp/newhosts /etc/hosts
I also recommend not only to use 1 hosts provider, but more and remove all redundant lines.
Well, I also don't recommend to cut "X" lines, cause it may vary from distro to distro...
If you wanna make it run periodically, just add on boot or make a .service
Hey, nice post... But I'd like to improve your script, if you don't mind...
Also, you should prevent detecing ad block with pinging the ads server, so it would be pretty to run a local server in 0.0.0.0:8080 (if this doesn't present any security risk).
I notice many folks move (mv) or delete the /etc/hosts file as part of the script or process.
If your machine is on-line, this can cause issues. Having worked on Servers, the /etc/hosts file should not be removed, the new file should be moved or copied into place. Make certain the permissions stay the same.
In a few scripts they:
cat [file] > /etc/hosts # has the same effect as copy.
cat [file] >> /etc/hosts # Appends the [file] to the target.
If you are really doing this on Linux you may be better off using something like dnsmasq or Marco Peereboom's AdSuck program or similar rather than putting into the hosts file.
But you can NOT really use the IPv4 address of 0.0.0.0 on Unix / Linux in a hosts file. The reason why is this IPv4 address works on Windows because it is Microsoft's IPC (InterProcess Communication) mechanism. It just sort of works on Windows but not on Linux. You can use my PERL daemon named phttpd that gives a 1x1 GIF image for images and blank lines but only on the loopback port (IPv4 127.0.0.1). This IS the IP address I use in my hosts file but I have an out for converting it to 0.0.0.0 on Windows that is the best option out there.
http://www.securemecca.com/phttpd.html
http://www.hostsfile.org/phttpd.html
Mine really is a daemon and does a proper double fork and setsid and after the great fracturing of Linux into umpteen different distros I no longer automatically start the daemon but just put phttpd in this folder:
/usr/local/sbin/phttpd
I start phttpd by first getting root running in a terminal. "su -l root" on RPM based Linux and "sudo su -l root" on Debian based Linux and then just do a:
sh
exec /usr/local/sbin/phttpd
ps -eadf | grep phttpd | grep -v grep # this shows phttpd if it worked
Alternatively you can use something like Cameleon's phttpd written in C:
http://sysctl.org/cameleon/
But first if you are going to put it in the hosts file you have to convert all of these 0.0.0.0 to 127.0.0.1. Here is the EASY way using sed (after you get to the folder where the new hosts file is at as YOU - do all work as you and then the final copy as root to /etc/hosts and I WOULD preserve the defaults that were in the /etc/hosts file at the start):
if grep '0.0.0.0' hosts > /dev/null
then
rm -f hosts.zero
mv hosts hosts.zero
sed s/0.0.0.0/127.0.0.1/ hosts.zero > hosts
fi
Henry Hertz Hobbit
(not anonymous after all)
PS I strongly urge the use of Alex Kowalski's hosts file maintenance program for Windows. It does automatic 127.0.0.1 to 0.0.0.0 conversion and lots of other nice stuff AND maintains the staticly set IP addresses for hosts you don't want to look up for DNS for what ever reason. I do it for DNS servers for example to make sure their IP address turns up in WireShark for example.
#!/bin/bash
#Just to make sure we can proceed safely
rm -rf /tmp/newhosts
mkdir /tmp/newhosts
cd /tmp/newhosts
wget https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
#remove all comments to optimize
sed '/^[[:blank:]]*#/d;s/#.*//;/^$/d' ./hosts
#try to preserve hosts entries
cat ./hosts >> /etc/hosts
#remove all duplicate lines
awk '!seen[$0]++' /etc/hosts
#remove unused files
rm -rf /tmp/newhosts
I liked what CaptainSkyhawk had but noticed a few things that could be improved on.
1.) I added the link to the wget line
2.) Looking for lines to start with 0.0.0.0 isn't enough. A malicious user could put multiple IP/host mappings on one line with &&. I added code to catch this.
3.) We might as well report which lines have problems, otherwise we have to go looking for them manually before we can fix them. Added code to print the offending info.
4.) I prefer hardcoding paths rather than using ~
5.) I don't see the logic in using 4 lines (cp, rm, mv, cat) when the same can be accomplished in 2 (cp, cat), so I cleaned that up.
6.) The anonymous poster pointed out the problem with catting with >> versus >, I corrected that as well.
Here's what I have. If anyone else can think of more improvements that can be made, please share them.
#!/bin/bash
# Backup current hosts file, cd to tmp, download new hosts file
printf "nBacking up current hosts file.n"
cp /etc/hosts /etc/hosts.orig
cd /tmp
printf "Downloading new Hosts file.n"
wget -q -O /tmp/hosts.new $remote_hosts_file http://winhelp2002.mvps.org/hosts.txt
# Remove commented lines and lines before hosts entries
egrep -v '^#' /tmp/hosts.new > /tmp/hosts.tmp
sed -n -i '/0.0.0.0 /,$p' /tmp/hosts.tmp
# Perform malice check to see if anything points to anything other than 0.0.0.0
if [[ $(awk '{print $1}' /tmp/hosts.tmp | uniq) != '0.0.0.0' || $(awk '/&&/' /tmp/hosts.tmp) ]]; then
printf "File HAS been tampered with. Non-approved value(s) found:nn"
awk '{print $1}' /tmp/hosts.tmp | uniq | grep -v 0.0.0.0
awk '/&&/' /tmp/hosts.tmp
printf "nExiting!n"
exit 1
fi
# If malice check is clean echo that and cat /tmp/hosts.tmp into /etc/hosts
printf "File has NOT been tampered with. Proceeding.n"
cat /tmp/hosts.tmp > /etc/hosts
printf "Done.nn"
Hi, I use a different method for loading a /etc/hosts file in one of my Puppylinux versions. There are some sites that use for example the Google captcha. I block Google in my modified hosts file, so I need to use the original, unmodified hosts file in such cases. I have copied the default hosts file, as 'hosts-orig', and I name the modified hosts file 'hosts-mod', both are placed in /etc/. I right-click the one I need for the moment, and choose to link it, as a relative link, which I name 'hosts', and just let it overwrite the old 'hosts' file. That works very well. When I need the other hosts file, I just link it, name it 'hosts', and just let it overwrite the old one again.
im hitting a snag on my pi install
dietpi@DietPi:~$ dietpi@DietPi:~$ sudo nano /root/update_hosts.sh
sudo: unable to resolve host DietPi: Name or service not known
dietpi@DietPi:~$ chmod +x update_hosts.sh
chmod: cannot access 'update_hosts.sh': No such file or directory
dietpi@DietPi:~$ sudo chmod +x update_hosts.sh
sudo: unable to resolve host DietPi: Name or service not known
chmod: cannot access 'update_hosts.sh': No such file or directory
dietpi@DietPi:~$ sudo chmod +x update_hosts.sh
sudo: unable to resolve host DietPi: Name or service not known
chmod: cannot access 'update_hosts.sh': No such file or directory
so then i tried modifying your tutorial.
dietpi@DietPi:~$ sudo chmod +x /root/update_hosts.sh
sudo: unable to resolve host DietPi: Name or service not known
dietpi@DietPi:~$ sudo chmod +x /root/update_hosts.sh
but same error.
your cookies police is a block to concentrate on the subject I am trying to concentrate!
I have no idea what this means.
Is this method still working? I mean it is more than 10 years old in the meantime? Thanks for the reply in advance! 🙂
Looks like they stopped maintaining it in March of 2021
just seems to be a "public forum" of work arounds to download a hosts file remotely off the web, and move it into /etc/hosts on a *NIX environment....I can't see this really "breaking" anytime.....? granted, some distros handle the hosts file different, but most are all in /etc/hosts
i dont think its working properly?
-2023-04-05 22:36:09-- http://winhelp2002.mvps.org/hosts.txt
Resolving winhelp2002.mvps.org (winhelp2002.mvps.org)... 198.187.28.133
Connecting to winhelp2002.mvps.org (winhelp2002.mvps.org)|198.187.28.133|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://winhelp2002.mvps.org/hosts.txt [following]
--2023-04-05 22:36:09-- https://winhelp2002.mvps.org/hosts.txt
Connecting to winhelp2002.mvps.org (winhelp2002.mvps.org)|198.187.28.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 334861 (327K) [text/plain]
Saving to: ‘hosts.txt’
hosts.txt 100%[=============================================================================================================================================>] 327.01K 866KB/s in 0.4s
2023-04-05 22:36:10 (866 KB/s) - ‘hosts.txt’ saved [334861/334861]
cat: /root/.etchosts: No such file or directory
dietpi@DietPi:~$ sudo bash /root/update_hostss.sh
and my bash file simply contains
GNU nano 3.2 /root/update_hostss.sh
#!/bin/bash
cd /tmp
wget http://winhelp2002.mvps.org/hosts.txt
rm /etc/hosts
mv hosts.txt /etc/hosts
cat ~/.etchosts >> /etc/hosts
on a raspberry pi
http://winhelp2002.mvps.org/hosts.txt "went off the air" because maintainer retired for health reasons...
I guess this is one of the pages that needs updating!
I wish the guy who main tain this starts it again. I just switched from Windows. I used it for the last 5 years on Windows.
You can visit https://github.com/hagezi/dns-blocklists