Github Repo and Usage Instructions
Abstract
I wanted to import Venmo transactions into my budgeting account, but Venmo doesn't make this easy for developers. So I created a email inbox and email-parsing bot to hack around it anyway.
The problem
I use You Need a Budget (YNAB) to keep track of all my spending. It's been incredibly useful, especially since I don't have a regular monthly income. Most budgeting tools are reactive, showing you where your money has been, but YNAB is proactive, asking you to plan and prioritize with the money you have available right now. It shows you where your money has been and asks you to plan what the rest of your money should do before you get more. Of course, that's most useful if you can track every dollar. And using Venmo makes that difficult.
Venmo isn't particularly friendly to developers. Most services expose an application programming interface (API). This lets developers ask the app for data or receive updates when an event happens.
It would be great if Venmo exposed such a service - my app could listen for transaction events from Venmo, then post them to YNAB. However, in 2016, Venmo halted access to its API for new developers.
Therefore, I needed a workaround if I wanted an automated way to get my Venmo transactions into YNAB. Entering them by hand is fine, but tedious. And what is a CS degree good for if not spending more hours automating than it would take to do the task by hand!
The Approach
I was inspired by the way Copilot handles Venmo transactions. They don't publish their Venmo integration, but they do tell you to set up an email rule that forwards all Venmo emails to a Copilot email address. This told me everything I needed to know: Copilot is treating the notification emails sent by Venmo as if they were webhooks from an API. They receive your forwarded email, parse it, and display your transaction in their app. Since YNAB exposes a public API, I am able to do the same thing! All I have to do it build it.
Building it
The code is available on Github, so I will remark upon some things that I learned while building.
I created a new Gmail account to serve as my parser email. This meant tangling with the Gmail API and Oauth credentialing. I used to manually update the Oauth credentials by going into the Google Cloud Console, creating a desktop credential, and changing my .env file. To make a more user-friendly credential refresh, I wanted to actually use a refresh token. I asked Claude to guide me through implementing an Oauth refresh flow, and it was instrumental. Now, the OAuth flow is initiated on the command line, and credentials are saved in a pickle file.
Once the emails have been retrieved, they need to be parsed. The emails that Venmo (and most other services) send are HTML emails, which makes them look pretty. That means I also needed an HTML email parser.
However, I learned that it was more complicated than that. Emails come from the server as a Message type object, which contains the raw email string among other information. I decode the raw bytes into a MIME message and save that to disk.
Then, to parse it from disk, I used the python Parser library to reconstruct an email object from the file saved on disk. Then, I fed that email object's payload to quopri to parse it. After these steps of decoding, the HTML can finally be read via BeautifulSoup. I used BS to retrieve all text strings from the email so I could extract the transaction informaton from them.
This is a bit silly? Why did I retrieve the emails, save them to disk, then use Parser, quopri, and BeautifulSoup just to get the email text? The short answer: testing. I wanted to modularize and each step of my code before I moved on to the next one. It took me a while to figure out how to authorize Google and retrieve the email messages, so I wanted to make sure I was retrieving them correctly before trying to parse them. Dumping them to disk seemed like the best way to do that. Saving the file to disk after each step also was helpful in determining what format the data was in. I've never worked with MIME types or Quoted-Printable encodings before. Having the file on disk was how I could inspect the data byte-for-byte to be able to determine the encoding.
To extract the transaction information from the email text, I employed a little library called parse. The parse library is the inverse of the string.format() python built-in, allowing you to create a format string and then check to see if a string matches the format. It's a slightly friendlier way than regex capture groups.
The Venmo notification emails follow a strict set of formats:
"{person} paid you ${amount}"
"You paid {person} ${amount}"
"{person} paid your ${amount} request"
"You completed {person}'s ${amount} charge request"To determine what type the transaction is, I compare the email's subject line to each of these formats and select the first (and only) valid one.
Next steps
I would love to make this something that the average YNAB user can use without needing to download anything, use the command line, or set up a cron job.
Security Concerns
If I did make this an app available to the public, I'd be concerned about the data privacy and security. Financial data is sensitive! However, emails themselves are plaintext and can be intercepted: not secure. Therefore, this data isn't as critical as something like credit card or social security information. Another concern is that I'd be assembling large amounts of data in a private email server. This data could be used deduce information about them such as spending habits.