Attempting to Block Hulu Plus Video Ads Using A Raspberry Pi
[UPDATE 2015-03-28] I have some more information about blocking video ads. I just need to make some time to write it up…
The keyword here is attempting. What is strange is that I actually had this working at one point. The Hulu video ads would be completely skipped (not just 30-90 seconds of silence). Unfortunately, when I reinstalled Raspbian and tried to follow my own steps (a few weeks later), it no longer worked. Since then, I have gone through the steps many, many times and have come to the conclusion that Hulu must have changed something on their end that disrupted my method, or I forgot to document a crucial step…
The basic concept was to deliver your own streaming content (stored on the Raspberry Pi) in place of Hulu’s video ads.
Disclaimer: It Worked Once, But I Cannot Reproduce The Results
As of now, this article is posted more for informational purposes as opposed to a working solution. I was able to completely skip the ads using this method, but could not once I followed through my own walkthrough. Maybe I missed something along the way, but I am hoping someone out there will know more than me and help figure out a way to block them.
Install With One Command (Someday…)
I also liked this idea so much, that I had it set up to be able to be installed with a single command. If I ever get it working, I will post that command here.
Conceptual Overview
When I had this working, it would use DNS spoofing and then deliver my own streaming content in place of Hulu’s video ads. The workflow went like this:
- If the DNS server running on your Raspberry Pi determines that the query is a URL known to serve ads, it redirects the client to the Pi instead of the real location
- A lighttpd Webserver running on the same machine (the Pi) then redirects that request to another URL, which delivers streaming content via minidlna .
In theory, everything sounds like it should work. Earlier, I speculated that the Hulu video ads might not be getting skipped because they needed to have streaming video content instead of a static Webpage. I still think this is true because I did have it working at one point. Give the steps below a try and let me know in the comments if you have any success. I am curious to know if anyone else can get it working.
Requirements For This Walkthrough
This is the culmination of the Pi-hole ad-blocker and is an fairly technical setup. As I mentioned, I had it working at one point, but when I tried to replicate it, I could not. I feel like Hulu might have changed something, or maybe a piece of the software used in this tutorial was updated/changed. Please follow the steps below and keep in mind that it might not actually skip the ads. Nevertheless, I wanted to post my efforts in case someone smarter knows what I might be missing.
Materials
- Local network
- Mac (or PC)
- Raspberry Pi set up as a pi-hole
Resources
- http://bbrks.me/rpi-minidlna-media-server/
- https://wiki.archlinux.org/index.php/MiniDLNA#Configuration
- http://manpages.ubuntu.com/manpages/precise/man5/minidlna.conf.5.html
- https://www.linode.com/docs/websites/lighttpd/lighttpd-web-server-on-debian-5-lenny
- http://redmine.lighttpd.net/projects/1/wiki/howtoredirectwww
- http://serverfault.com/questions/105920/how-do-i-redirect-multiple-domains-to-a-single-domain-in-lighttpd
Downloads
How To Block Hulu Plus Ads On A Raspberry Pi
If you already have set up the Raspberry Pi as an ad-trap, you might just need to modify some of the steps below but this walkthrough can be done as a stand-alone project, so you may want to start from a fresh, clean install of Raspiban. You will need both dnsmasq and the lighttpd Web server for this walkthrough, as well as minidlna .
Conceptual Overview
- Set up your Pi as a DNS server
- Set up your Pi as a Webserver
- Set up MiniDLNA to serve streaming video content to your devices
- Upload a very short video to the Pi that will be shown instead of the Hulu ads
- Re-direct Hulu’s ad URLs to the page hosting the video uploaded in step 2
- Set your device to use the Raspberry Pi as its DNS server
- (Almost) never see an annoying video ad again
Install dnsmasq
Run these commands to install dnsmasq
sudo apt-get -y install dnsutils dnsmasq sudo service dnsmasq stop
Configure DNS
Make a backup of the original dnsmasq.conf file so you have something to revert to if you make a mistake.
sudo mv /etc/dnsmasq.conf /etc/dnsmasq.conf.orig
Then, edit a new config file with the following content (note that this is slightly different than the previous setups):
sudo vi /etc/dnsmasq.conf
no-resolv no-poll # Change the IP below to the IP of your Raspberry Pi server=/localnet/192.168.0.1 server=/localnet/8.8.8.8 server=/localnet/8.8.4.4 domain-needed interface=eth0 strict-order min-port=4096 cache-size=10000 log-queries bogus-priv
To see what some these options mean, check out the original Pi-hole ad blocker.
Configure the Hulu Ad URLs
Next, create a list of known ad URLs that will be redirected when queried by DNS.
sudo vi /etc/dnsmasq.d/huluAdList.conf
You can add two different things to this file. If you strictly want to block Hulu video ads and nothing else, make a file with just the Hulu ad URLs, which I will discuss in a moment. Or you can run the gravity-adv.sh script, which pulls ad URLs from many different sources (including many Hulu ones). Note that this script will create a file called adList.conf instead of huluAdList.conf . Both files should be able to co-exist, but I don’t know what happens if there are duplicate entries in each.
So according to Hulu’s information, in order for ads to get through, the following two domains need to be unblocked:
- p.hulu.com
- ads.hulu.com
I set up my Pi-hole to block just those two domains, and I can see that it is actually serving up my streaming video instead of the ad image as you can see in the screenshot below:
So I know that the Pi-hole is working, but when you actually try to do this while streaming a video, the effect doesn’t quite work. On a few occasions, I saw the Pi-hole image for a moment and then it would disappear. I know I am close to a solution, but just can’t figure out that last piece of the puzzle.
If you actually watch the log file when streaming a Hulu video, you will see that there are a lot more URLs being queried then just the two Hulu tells you about. My guess is as good as anyone’s on which ones need to be blocked. Here is a list of ones that I think are ad URLs.
a.dlqm.net ads-a-darwin.hulu.com ads-bid.l.doubleclick.net ads-v-darwin.hulu.com ads-v-darwin.hulu.com.c.footprint.net amch.questionmarket.com assets.huluim.com bs.serving-sys.com cdn.innovid.com core.insightexpressai.com dart.l.doubleclick.net data.agkn.com ds.serving-sys.com dts.innovid.com googleads4.g.doubleclick.net hulu-darwin.fms.c.footprint.net hulu-pubnet.telemetryaudit.com p.hulu.com pagead.l.doubleclick.net pagead.l.doubleclick.net s.innovid.com sb.scorecardresearch.com secure-us.imrworldwide.com stats.l.doubleclick.net t.hulu.com t2.hulu.com tapestry.tapad.com telemetry.huluad.com ts.vindicosuite.com view.atdmt.com.nsatc.net vindicoasset.edgesuite.net
The /etc/dnsmasq.d/huluAdList.conf will need these domains to be in the format:
address=/domain.com/192.168.1.101
Just change the IP address to the address of your Raspberry Pi. See an example of this file here.
Install MiniDLNA
Install MiniDLNA
sudo apt-get install minidlna
Create A Folder To Store The Pi-hole Video
sudo mkdir -p /var/lib/minidlna/videos
Edit the MiniDLNA Config File
Make a backup of the config file in case you mess up:
sudo cp /etc/minidlna.conf /etc/minidlna.conf.orig
and then run this command to see what is currently enabled in the config file (it might be nothing):
cat /etc/minidlna.conf | grep -v "#" | sed '/^$/d'
Edit the file (uncommenting items, if necessary) similar to the following. You can run the command from above to verify everything matches. Alternatively, you can just create a brand new file with the information below instead of finding and uncommenting features.
sudo vi /etc/minidlna.conf
media_dir=V,/var/lib/minidlna/videos/ port=8200 friendly_name=pihole serial=12345678 model_number=1 inotify=yes
Below are what each of the options mean:
- media_dir=: this is where the video files are stored. The capital “V” tells minidlna that this folder will only contain video files. You can also add “A” for audio, or “P” for pictures. The letter is followed by a comma, and then the path to the folder.
- port: port to access from a Webpage
- friendly_name: this will show up in VLC so it is easy to find
- serial: reported to clients (might not even be necessary)
- model_number: also reported to clients
- inotify: watches paths for changes (additions of files)
Start the streaming service
sudo service minidlna start
You can verify the streaming service is working by navigating to the Pi’s IP address at port 8200 :
http://192.168.1.101:8200/
Copy A Small Video File Over To The Raspberry Pi
There is nothing on the server yet, as seen in the screenshot above, so we need to add a file. For this project, we only need one. It will be a very short video that will take the place of the Hulu ads.
Put The Video In A Folder So MiniDLNA Can Find It
The easiest way to get this file is to run a curl command from the Pi.
sudo curl -o /var/lib/minidlna/videos/pi-hole.mov "https://dl.dropboxusercontent.com/u/16366947/Files/pihole.mp4"
This will just download it directly to your Pi in the correct folder.
If you have the file on another local machine, copy the pi-hole.mov over to the Pi using scp . To send the file to the pi from another computer use this command.
scp ~/Downloads/pi-hole.mov pi@192.168.1.101:/var/lib/minidlna/videos
to get the file (while on the Pi), use this command
scp <YOUR_-- USERNAME>@<IP_ADDRESS>:/Users/<YOUR_-- USER>/Downloads/pi-hole.mov /var/lib/minidlna/videos
Reload MiniDLNA
sudo service minidlna force-reload
Check that it found a file by running this command:
tail -f /var/log/minidlna.log
You should see an entry showing that it found a file (highlighted below)
....... [2014/10/11 03:59:27] minidlna.c:1006: warn: HTTP listening on port 8200 [2014/10/11 03:59:27] scanner.c:727: warn: Scanning /var/lib/minidlna/videos [2014/10/11 03:59:28] scanner.c:798: warn: Scanning /var/lib/minidlna/videos finished (1 files)! [2014/10/11 03:59:28] playlist.c:125: warn: Parsing playlists... .......
You can also check the Webpage again to see if it found a video file. If you used a different video than the one provided in the Downloads section (or from the curl command), and it didn’t show up, it might not be formatted correctly or it might be too big. I originally tried it with an MP4 from Final Cut Pro X, but it didn’t work. I exported the same movie as the lowest resolution possible and it worked.
Find The URL Of The Video
- Open VLC and click Universal Plug ‘n Play (UPnP)
- The expand pi-hole > Video > All Video > pi-hole.mov
- Right-click it and choose Media Information.
This will show you the URL of the video file. This URL will be needed in the lighttpd config file (this will serve requests for Hulu ads with the Pi-hole video instead of the ad).
Configure lighttpd To Redirect Hulu Ads To The Pi-hole Video Instead Of Your TV Screen
This is where the magic is made. Edit the lighttpd config file. We are going to tell the Webserver to redirect any Hulu Plus ad URLs to the video we uploaded earlier.
sudo vi /etc/lighttpd/lighttpd.conf
Make sure mod_redirect is uncommented:
server.modules = ( "mod_access", "mod_alias", "mod_compress", "mod_redirect", # "mod_rewrite", )
At the end of the config file, you will need to add entries similar to the following. One for each Hulu ad URL you intend to block with a pipe | in between them. Your URL from before goes on the redirect line in between the quotes.
$HTTP["host"] == "ads.hulu.com|p.hulu.com" {
url.redirect = (
"^/(.*)" => "http://192.168.1.101:8200/MediaItems/19.mov",
)
This tells lighttpd that if someone is asking for the content at ads.hulu.com or p.hulu.com, they should instead get redirected to the movie file running off of MiniDLNA.
The /usr/local/bin/gravity.sh script that I made will add a bunch of Hulu urls to the list. You can see some of them by running the file gets created.
cat /etc/dnsmasq.d/adList.conf | grep hulu
This will return a list of “known” Hulu ad URLs that are already set to be routed to the Raspberry Pi instead of the real location.
address=/ads.hulu.com/192.168.1.101 address=/ads-v-darwin.hulu.com/192.168.1.101 address=/a.huluad.com/192.168.1.101 address=/ads.hulu.com/192.168.1.101 address=/ads-v-darwin.hulu.com/192.168.1.101 address=/a.huluad.com/192.168.1.101 address=/hulu.112.2o7.net/192.168.1.101 address=/huludev.112.2o7.net/192.168.1.101 address=/ll.a.hulu.com/192.168.1.101 address=/t2.hulu.com/192.168.1.101 address=/t2.huluim.com/192.168.1.101 address=/t-ak.hulu.com/192.168.1.101 address=/track.hulu.com/192.168.1.101 address=/tw.i.hulu.com/192.168.1.101 address=/urlcheck.hulu.com/192.168.1.101
Some Hulu Video Ads Are Still Getting Through
The list above is not a be-all end-all aggregation. The ads also get served from other third-party sites that may not appear on the list. If you watch the dnsmasq log file while an ad is playing on Hulu, you will see all the URLs being queried.
tail -f /var/log/daemon.log
From there, you can start appending these URLs to the /etc/dnsmasq.d/adList.conf list. If you don’t add them here, DNS won’t know to route them to the Raspberry Pi instead of the real location. But, you also need to add the domain to the /etc/lighttpd/lighttpd.conf file. Both parts are necessary for the video ads to be neutralized.
Testing Redirection
You can test if your redirection is working with a Terminal command from a Mac on the same network (make sure the Pi is set as this computers DNS server).
curl -I ads.hulu.com
If it is set up correctly, you will see output like this:
HTTP/1.1 301 Moved Permanently Location: http://192.168.1.101:8200/MediaItems/19.mov Date: Wed, 19 Nov 2014 00:11:52 GMT
If it is not working, you will see
HTTP/1.1 403 Forbidden Server: nginx/1.4.2 Content-Type: text/html Content-Length: 168 Cache-Control: max-age=31536000 Date: Tue, 18 Nov 2014 15:13:38 GMT Connection: keep-alive
If you were able to skip the Hulu ads, please let me know in the comments what you did differently.