Using django-paypal

Today I decided to do some simple paypal integration for a personal site of mine.  From a (very) quick search, it looked like django-paypal over at github (https://github.com/johnboxall/django-paypal) was the most recent app providing Paypal helpers for a Django project.  I was looking for a simple "Buy Now" solution where the user is taken to paypal's site to complete the payment and my site gets an IPN (instant payment notification) from paypal when that is completed.

django-paypal looks to be simple to setup, and for the most part this was true. I did hit a few snags during setup and initial testing with the paypal sandbox, which I'll describe below.  Skip to the end if you just want the quick and dirty version.

I followed the instructions in the README and I will be referencing the various steps there.  Step 1 involved cloning the git repository, adding 'paypal.standard.ipn' to my INSTALLED_APPS and setting the PAYPAL_RECEIVER_EMAIL setting.  Then I added a payments app to my project with a simple view which rendered an instance of the included PayPalPaymentsForm (see step 3 in the README).  I made a couple of mistakes when setting up the paypal_dict to use with the form: I used my live email address, and I left the 'invoice' parameter set to 'unique-invoice-id'. More on these later.  I also changed the form template to use "{{ form.sandbox }}" instead of "{{ form.render }}".  These custom methods output a full HTML form and setup the action to submit to the paypal sandbox and live site, respectively.


In step 4 of the README, I setup a URL to handle IPNs.  Not thinking, I made this different than the "notify_url" parameter I had set in the paypal_dict in step 3.  Don't do this -- "notify_url" needs to hit the url you specify in step 4.

And step 5 was just setting up a signal handler for when the IPNs are processed. I connected only to the payment_was_successful signal.

Now, onto the problems I ran into.

Of course, I wanted to test my implementation using the sandbox, so I signed up for a developer account and created a buyer test account.  I then started the checkout process by hitting the view I had configured in step 3, and signed into the paypal sandbox.  When I tried to pay, I was notified that the invoice had already been paid for and was unable to continue.  I remembered the "invoice" setting in step 3 and went back and customized it to something unique.  Retrying the checkout let me progress further and I successfully completed payment.

However, watching the development server showed me that paypal was posting to a URL didn't exist since I had set "notify_url" to something different from the url pattern in step 4.  I corrected that mistake and tried again.

This time, after logging in and clicking "pay", I got a very nasty paypal backtrace of sorts. After a while of debugging, I noticed that this error appeared to go away if I changed the configuration form being submitted to paypal, via the paypal_dict.  Namely, I seemed to have to change the "invoice" parameter every time -- which is strange since I assumed I would just get the "invoice already paid" error.

Having figured that out and being able to get paypal to tell me that payment was successful, I turned my attention to the IPN I should be receiving but wasn't.  The development server was having issues with the POST paypal was making for the IPN.

Traceback (most recent call last):
  File "/usr/local/lib/python2.6/dist-packages/django/core/servers/
basehttp.py", line 281, in run
    self.finish_response()
  File "/usr/local/lib/python2.6/dist-packages/django/core/servers/
basehttp.py", line 321, in finish_response
    self.write(data)
  File "/usr/local/lib/python2.6/dist-packages/django/core/servers/
basehttp.py", line 417, in write
    self._write(data)
  File "/usr/lib/python2.6/socket.py", line 300, in write
    self.flush()
  File "/usr/lib/python2.6/socket.py", line 286, in flush
    self._sock.sendall(buffer)
error: [Errno 104] Connection reset by peer 


After a bit of googling, it appeard that paypal might be trying to connect to my development server using HTTPS, which is not supported.  The stunnel package was suggested as a possible fix for this during development. After just a little wrangling, I managed to get stunnel to accept the HTTPS connections on port 8000 and then "forward" (not sure of the correct terminology here) the contents -- as normal HTTP requests -- to the development server on port 8080. I had to change the 'notify_url' parameter to use the new port after setting this up.

stunnel -d 0:8000 -r localhost:8080


Now the paypal POSTs were getting to the dev server, but were being rejected with a 403 (forbidden) status.  I think most djangonauts immediately recognize this as a possible CSRF-protection issue.  Out of the box, Django protects all POSTs by requiring a cryptographic token to prove that the request is from a valid source.  Normally you would provide this token in a form being submitted from and to your own site.  Since paypal is generating these POSTs for the IPN, I (probably) have no way to give them such a token and so Django rejects those requests.  The simplest fix here was to modify the source of 'paypal.standard.ipn.views' and decorate the 'ipn' view there with  'django.views.decorators.csrf.csrf_exempt'.  I haven't investigated but there may be a way to give paypal a custom key,value pair to include with the IPN, in which case you could probably generate a CSRF token and remove that 'csrf_exempt' decorator.

So, now the IPN requests are getting to the dev server which is responding with HTTP 200 (a good thing), but I'm still not seeing my signal handler get run.  A quick check of the admin interface shows that the incoming IPNs are all flagged with "Invalid payment_status. (Pending)".  This seemed strange: why wouldn't the payment be completed immediately?  I logged into to the test user's account and noticed that all of the transactions were "unclaimed".  Again, I was confused until I remembered that I had configured the 'business' key in the paypal_dict (step 3) to my live paypal account.  That email address does not have an account in the paypal sandbox world, so the payments are "pending" until I sign up and claim them.   I went back to the paypal developer console and added another test account, this time a seller account.  I copied the resulting email address into the 'business' key for the paypal_dict and hoping that was the end of it.

The next attempt still yielded a flagged IPN, but the error was different.  This one said "invalid receiver_email." Another quick scan of the django-paypal code showed that it compares in the incoming IPNs to the PAYPAL_RECEIVER_EMAIL setting I configured in step 1.  Again, I used my live email address here.  After changing that to the test seller account, the stars aligned and angles sang.  The next payment attempt came through successful and not flagged.

To summarize, here are a few of the things I had to do in order to get django-paypal working in test mode with the sandbox:

• Use a new 'invoice' for each attempt
• Match the 'notify_url' with the IPN url pattern.
• Use stunnel or an SSL-enabled webserver to handle IPN requests.  Some people say setting the protocol on 'notify_url' to 'http' is enough to tell paypal to not use HTTPS, but it didn't seem to be true in my case.
• Make sure the provided 'ipn' view is CSRF-exempt or find a way to give paypal a CSRF token and have them include it in the IPN request.
• When testing, make sure you set both PAYPAL_RECEIVER_EMAIL and the 'business' parameter to a test seller account.
• In any case, make sure PAYPAL_RECEIVER_EMAIL and the 'business' parameter match. I just imported settings and set 'business' to settings.PAYPAL_RECEIVER_EMAIL instead of redefining the email address in both places. DRY.

django-inotifier development story

The other day I started a new Django project which required me to know when  files were received by dcmrcv -- a file transfer agent in the DICOM Toolkit (dcm4che.org).  One way to accomplish this would be to monitor and parse the output of dcmrcv, perhaps with a hairy regex.  I decided to avoid that punishment and instead settled on using inotify, or more specifically, pyinotify, to monitor the directory and notify me when a new file appeared.

Background from wikipedia:

inotify is a Linux kernel subsystem that acts to extend filesystems to notice changes to the filesystem, and report those changes to applications.

For this project I just needed to setup pyinotify to watch a single directory -- incoming/ -- and handle the signals that it generates when file events happen there.  There are a number of different events that pyinotify can send but I only needed two -- IN_CREATE and IN_MOVED_TO.  I started just watching for IN_CREATE, but dcmrcv did things a little differently and required a little more work.

When receiving a file, dcmrcv creates a new temp file ending with a .part extension and writes chunks to this file as it receives them from the network.  When it has received everything and written it, it moves the .part file to the final filename.  This move does not generate a IN_CREATE event from pyinotify even if the final file did not exist beforehand.  It does, however, generate an IN_MOVED_TO event so I combined the monitoring of both to track a file from .part creation to the move to the final filename.  This is the CreateViaChunksSignaler() class in inotifier/event_processors.py.

I now had my event processor class, the events I needed to watch for, and the path to watch.  Now I just needed to hook it all into my Django project in a convenient way.  The end goal of this was to create a model instance for each file received by dcmrcv, so I considered just sticking all of this pyinotify stuff into the app where that model lived.  I didn't really like that, as the functionality was necessary to accomplish the task but it wasn't specifically relevant to that app.  At this point, I realized I could spin-out the filesystem monitoring stuff into its own more generic application and connect to it via signals.  The perfect recipe for a pluggable app!

I wrote a couple fairly simple management commands to start and stop the pyinotify daemon, added a Django signal for each pyinotify event, and settled on a single setting needed to configure it for each project.

The end result is a pluggable app which allows you to monitor any number of paths on your filesystem.  A different set of events can be watched for on each path, and they can be handled in different ways by using different event processor classes.  For a quick start, you can use the included AllEventsSignaler() class and just connect to the signals which you need.  Enjoy!

Announcing django-inotifier

I recently published my first open source project (BSD Licensed), django-inotifier.  The short description I've settled on is "a pluggable app providing a thin django wrapper around pyinotify".  It provides the following: Django signals which map to pyinotify events, management commands to start and stop the pyinotify daemon, and a few example event processor classes which send the signals.  It requires a single setting for configuration.

The idea is that pyinotify is used to watch a set of paths for file-system changes.  When those events happen, they are handled by an event processor class, which sends corresponding Django signals.  Other apps can connect to those signals and do Useful Stuff ™.

django-inotifier is available on github at http://github.com/jwineinger/django-inotifier