Reuben Castelino

This is my follow-up article to Replacing Google Reader with TT-RSS on Heroku. That article was just my quick notes on how I had set up tt-rss on Heroku as a quick transition from Google Reader. As with any project, nothing is ever perfect the first time and there were a lot of updates that I did until I was comfortable with the setup. Besides some small changes and bug fixes, the biggest two events would be the creation of a installation script and the addition of a self-updater. If I come across anything else, I’ll either post it at the end of this article or write a Part 3. I’m still debating.

Cost

I was indirectly alerted by Survector in his comment that I might not have fully explained what Heroku was, its pricing, and its pros and cons. Heroku is a PAAS (Platform As A Service) which allows developers to host their applications under free or premium accounts. Primarily used for Ruby on Rails projects, Heroku has support for many languages (and unofficial support for PHP which was used for ttrss). Each time a Heroku application is created, the developer is basically given a small computer that has 512 mb RAM running Ubuntu on which the application is run. This computer/server is called a dyno and it operates on a sequential basis meaning that it can only only task at a time. In my ttrss server, I created a Heroku application and gave one dyno the task of running the tt-rss server. Because I was only using one dyno, I was able to stay on the free tier: there were NO costs to run this server. This was nice, because the hosting was taken care of with no costs. Because it the server was only used by me, the dyno could handle the load and I had no need to upgrade.

Emulating Google Reader

Thanks to #6 on Jan’s post, I was able to emulate the posts as it appeared on Google Reader where by clicking on an article, it would expand to show the feed article. This combined expandable view was much better than the default split view. This can be enabled in the preferences section. I have also started changing the css with my own custom css to have bigger text and more whitespace. There are plugins/patches on the tt-rss forum if you want to use a shortcut or you can use the customize button in the preferences. I first used the live css editor (available in chrome or firefox) to change up the layout and then copy and pasted it into the customize css option to save them. If you want a tutorial, ask in the comments and I’ll eventually provide one.

Shell Script

Inspired by a post on the tt-rss forum in which who_me created a script that allows easy installation of tt-rss on openshift, I decided to make it easier to create tt-rss servers on Heroku. Hosted on github, the script is pretty stable (hopefully). It’s just a bash script that runs through all the commands and should successfully create a server for you. I’ll occasionally update it with the correct version number. It sets up the server and a self-updating feature which will be explained later on.

Procfile and web-boot.sh (bug)

For some reason, the default httpd.conf for PHP files has a MaxClients of 1 meaning that only one person can connect to the server. Obviously, this can be a problem. So I had to change it using a custom worker. Workers are another name for the task that is assigned to a dyno. So I created a file called Procfile in the top directory and in it inserted this line

web: sh www/web-boot.sh

This tells the Heroku dyno to run the web-boot.sh file instead of the default boot.sh file on Heroku’s root server. So obviously, the next step would be to create a web-boot.sh script. In the same directory as the Procfile, create a web-boot.sh file and put these lines in it:

sed -i 's/^ServerLimit 1/ServerLimit 8/' /app/apache/conf/httpd.conf
sed -i 's/^MaxClients 1/MaxClients 8/' /app/apache/conf/httpd.conf

sh boot.sh

This tells Heroku to up the ServerLimit to 8 as well as the MaxClients. After that, run the default boot.sh file to start tt-rss. Pretty easy so far.

Self-Updating

The problem here is that tt-rss doesn’t have a real-time updating solution like google reader did. This is basically because constantly polling for updates would take up a lot of resources that a self-hoster doesn’t want to do. And so tt-rss has a update.php script that can be run to update all feeds. tt-rss doesn’t have a built in update-all button, because if someone has a lot of feeds, it would take too long. For example, I have a little over 50 feeds and just importing them and updating them initially all took ~10 mins. Right now, after syncing it all, my feeds update at about a rate of 1/2 a second per article. This can be a problem to update over http because most servers (like Heroku) have a 30 second timeout. So while Jan’s post (mentioned above), does mention a url to update, Heroku times out and crashes the application. And when it crashes, it stays crashed for at least 5-10 mins-which is unacceptable. Previously, I had a cron job run every 5 mins to update the feeds, but if my computer was off, nothing would update and if I’m gone from home, I have no way to update everything.

But I couldn’t get Heroku to update the tt-rss server either. Heroku only allows one dyno per application on the free tier. If I had a worker dyno (background jobs) to my web dyno (application job), it would bump up the price to $34 per month which was way to much. And so I threw every idea I had at it, but nothing would work. The one idea I had that seemed the most profitable was to create another Heroku application that could somehow interface with my tt-rss server and remotely update it, but I had no idea how to do this.

After a couple of days of wrestling with it, I came across this post by arzumy which fully solved my problem. Basically, you have to create two applications in the same folder so that each folder. First create the first application like how we did in Part 1. Remember to use the Procfile and the web-boot.sh file this time instead. However, before you commit the changes to Heroku, add to the Procfile this line:

worker: while true; do ./php/bin/php -c www/php.ini ./www/update.php -feeds; sleep 300; done

This adds a worker dyno which will run the update script every 5 mins. But someone smart is yelling at me, “Hey! You can only have one dyno!” And that person would be right. So here’s how to get around it. Push what you have to your heroku application so that works. Go to your tt-rss server online and make sure it works. If it does, we’re ready to move on.

In the same folder as your application, create a new application like this

heroku apps:create appname-updater

Now you need to connect your new application to your old one. Forewarning, because you now have two applications created in the same folder, you always need to specify which app you’re talking about with “–app appname. Grab the database url for the tt-rss server with

heroku config --app appname

Copy and paste the database url into your new updater application as well as adding the proper environment variables.

heroku config:add DATABASE_URL=databaseurl --app appname-updater
heroku config:add LD_LIBRARY_PATH=/app/php/ext:/app/apache/lib --app appname-updater

Now add the application repository to your git config with

git remote add appname-updater git@heroku.com:appname-updater.git

Now push your data to your second application with

git add .
git commit -m "initial commit"
git push appname-updater master

You might also want to push all your data to your ttrss server just in case with the same command as above just replacing the appname-updater with heroku.

Now you need to assign your workers. Your first application will still only be using the web dyno as it was originally. However, Heroku has this ability where you can have one worker dyno instead of one web dyno and it is still totally free. So assign a worker dyno to your heroku config with

heroku ps:scale worker=1 --app appname-updater

If everything worked out well (and I didn’t forget anything), you should now recieve updates every 5 minutes. You can check this by running

heroku logs --app appname-updater

New Versions

So updating to the next version basically sucks. I’ve updated the scrip to 1.7.5 so newcomers should have no problems. For oldies who worked with previous versions, we have it tougher. There are two ways to do it. The easier way is to export your feeds, delete your apps in heroku, and recreate it with the script, and then importing it. However, if you’re sentimental like I am, and want to stick with your original setup, here’s how to do it.

First make sure that you have php and postgresql installed on your computer. The php installation must be able to work with the postgresql installation as well. In arch linux, its as easy as installing php-pgsql and uncommenting the necessary line in /etc/php/php.ini.

Then in your local tt-rss installation, in your config.php add updater to your plugins section near the end of the file. After that, run

/usr/bin/php update.php -update_self

It should update your installation and create a folder up one directory of your old config. USING A SUDO COMMAND, transfer your git folder back into your tt-rss folder. This is important so that you don’t lose your important .git setup. Then run

sudo chown -R localuser:users .git

so that you can edit the files like normal. Transfer your Procfile, web-boot.sh file and as well as any other files you might have edited. Then run git status to see all the changed files. Run

git add .
git add -u

and then

git status

to see if there are any more changes to be committed. That should be all of them. Then commit them

git commit -m 'updated to 1.7.5'

and push it

git commit heroku master
git commit appname-updater

I was definitely tired after trying to figure this out, and blasted through this walkthrough. If I missed anything, let me know and I’ll fix it. This is such a precarious update that I don’t want to try and automate it. If anyone can figure out how to successfully, I’ll be happy to include it in the ttrss-on-heroku repo for others to use.

Ironically, even though I was sentimental and wanted to keep my old setup, I screwed this section up so often, that I had to recreate my ttrss server ~5 times before I figured it out. So much for keepsakes.

All Finished

Now, I have a tt-rss server that self-updates and provides rss feeds for me whenever I need them. Besides, this version updating (which is technically not even necessary), I can consider this project sufficiently completed.

I’m pretty sure that I’ve cited all my external sources in the text above, but nevertheless: thanks to all who provided me with information and tutorials especially arzumy who in a couple of paragraphs solved all my updating woes.