Posted: Mar 25, 2010
by Billy Gray
Tagged strip, beta, stripsync, import, csv
Updated: 5/5/2010 Bulk updating via CSV import is now supported in Strip Sync for Mac OS X (SSM), article below has been updated.
As mentioned here previously, we’re on the cusp of starting the beta test of Strip Sync, our new desktop companion tool for Strip. Among the application’s features are CSV import and export. The intent of this article is to describe the format used on export and required on import. This format is subject to change, but it won’t be changing much. Any changes will be announced here on this blog and noted as updates at the end of this article.
The General Gist
Each row in the CSV corresponds to an Entry in your Strip database. There’s a field indicating what Category the Entry belongs in, the name of the Entry, and every other column is considered a Field. The import process creates a new Entry for each row in the CSV file after the header row. Note: Bulk updating is now supported on SSM, requires use of EntryID column. Bulk updating will be made available soon in Strip Sync for Windows (SSW).
This is what a sample import CSV might look like:
Entry,Category,Account,Email,Note,Password,Phone,PIN,show,Username,Website
Credit Card,Financial,3759 876613 21001,,"exp:12/12
CVV:3829",secret,1-800-123-4567,4,,mscott,http://mycreditcard.com
Insurance Policy,Financial,3759 876613 21001,,secret: name of your first pet? spot,secret,1-800-123-4567,4,,mscott,http://myinsurance.com
jordie laforge,trekkers,,,,,,,nextgeneration|deep space nine,,
kirk,trekkers,,,,,,,star trek,,
patrick stewart,trekkers,,,,,,,star trek|next\|generation|voyager,,
riker,trekkers,,,,,,,,,
Shopping Website,Personal,,mscott@mailinator.com,,secret,,,,,http://paypal.com
Header Row Required
Just like the subtitle says, a header row is currently required to describe the data in the spreadsheet you’re importing for Strip.
Header Specification
- One column must be named “Entry”, and this is case-insensitive.
- One column must be named “Category”, and this is also case-insensitive.
- The name of every other column (for now) is considered the name of a Field.
No columns should be named “Guid”… Guid is no longer a restricted name.- One column may be named “EntryID” on SSM, making the row an update to an existing record.
When Strip Sync reads the header row of your import file, it looks up each Field name in your database to see if there’s already a label/type associated with it. If not, a Field with this label is created for you in your database, with the default mode set to “text”, and you can simply change this setting to URL or whatever you like by editing your labels in Strip.
When a row contains an EntryID, Strip Sync looks up the Entry in your database and replaces it’s name, category, and fields using the data in the rest of the row.
Bulk Updating
To do bulk updates via CSV import, you need to get the unique identifiers for your entries! Simply use the export feature of Strip Sync to export a CSV file containing all records in your database, with their EntryIDs.
Row Processing
During import, after the header row has been read, Strip Sync begins cranking through all the other rows, creating new Entries using the data in each row (bulk update via CSV import is not supported yet, but we plan to support that soon). Here’s how it works:
- Strip Sync looks at the Category field and does a case-sensitive lookup to find a matching Category in your database. If no match is found, a new Category with this name is created.
- The Entry column is used as the name of the new Entry, as indicated above.
- For each additional column in the row:
- If the column is empty, it is ignored
- If the column is not empty, a Field is created on the Entry, with a type/label corresponding to the column’s header name.
- Field columns may contain multiple values, separated by the ‘pipe’ character, ‘|’. If multiple values are detected, multiple Fields will be created on the Entry, labeled according to the column’s header name.
- If your Field needs to contain a pipe character as part of the Field value, you may escape it with a backslash character (i.e. ‘\|’).
- If an Entry or Category column is blank, the entire import will be rolled back, and an error message will display detailing the problem and the line number where the problem was found.
Our import and CSV processing is based on the scanning technique and EBNF outlined by Matt Gallagher to fully support properly escaped CSV data.
If one were to extend that EBNF definition to take into account our use of | to separate multiple field values, we think it would look like this:
file = [header lineSeparator] record {lineSeparator record}
header = name {separator name}
record = field {separator field}
name = field
field = escaped | nonEscaped
escaped = doubleQuote {innerField | separator | lineSeparator | twoDoubleQuotes} doubleQuote
nonEscaped = innerField
doubleQuote = '"'
twoDoubleQuotes = '""'
separator = ','
lineSeparator = ('\r' | '\n') {'\r' | '\n'}
innerField = textData { innerFieldSeparator | textData }
innerFieldSeparator = '|'
textData = {characters up to the next double quote character, un-escaped innerFieldseparator, separator string, or lineSeparator}
Obviously, commentary and corrections are welcome (as well as bug reports).
Posted: Mar 24, 2010
by Billy Gray
Tagged tempo, update, time, tracking
Last night I pushed a couple of updates out to our Tempo time tracking service that were a touch over-due:
- Tweaked date field on time entry to fix text-obscuring problem
- Tweaked order receipt for providing copies of past invoices
- Improved navigation and visibility for Team → Status
- Fixed display margins in module boxes site-wide
- Adds a “add new user” link in project team settings for convenience
- Allows meta-characters in tags, no longer restricted to [A-z0-9-_]
The tags thing was getting annoying, I will admit. It’s pretty useful to be able to, for instance, begin certain tags with a * character to mark them as project defaults.
The Team Status thing was a visibility issue. I think most administrators and managers of Tempo accounts were unaware that in addition to the basic Team view (a simple list of your users), there’s a Status view indicating who is working on what:
Aaaaand it even has a handy mobile view that you could bookmark:
If you notice any bugs, or there’s anything bothering you about the system, please get in touch.
Posted: Mar 23, 2010
by Stephen Lombardo
Tagged jobs
We’ve been really busy here at Zetetic lately and we’re looking for a new teammate to help us develop and support client systems and our own products. As a small and flexible software development consultancy, we’re mostly interested in personal qualities, not resume buzzwords:
- Natural or Artificial intelligence (robots welcome)
- Intense desire to learn new things & hack on technology
- An unflappably positive attitude
- Plays well with others
A background in one or more of the following goes a long way too!
- Strength in at least one Object Oriented Programming Language (C#, Ruby, Java, Obj-c). Anything except VB.
- Desire to “switch things up” and work on multiple programming languages
- Experience with databases, relational SQL, or otherwise (mongo, couch, etc)
- Security Technology (i.e LDAP authentication, Single Sign-on, PKI)
- Linux and general networking
- 2-5 years of software development experience
- Located in Philadelphia, NYC, or Central NJ to allow collaborative work with our current team members
In short, we mostly care that you’re smart, love to hack, and get things done.
If this sounds like you, or someone you know, please reach out and let us know: sjlombardo@zetetic.net.
Posted: Mar 19, 2010
by Billy Gray
Tagged strip, iphone, sync, stripsync, macosx
I have a couple small things to fix still, but we’re planning to start the beta for Strip Sync next week. We’ve been getting a lot of “where is it!?!” emails, so we thought it’d be a good idea to post some screens and show you that it’s not vaporware. I’m heading development on the Mac side, so I’ll show you screens from Mac OS X. You’ll get a look at the .NET version for Windows soon.
When you start up the Strip Sync application, it is locked, and requires you to enter your access password:
On first-time start-up, it will ask you to set a password for the local database replica. I should note that, at least for the time being, the password on your desktop database must match that of your iPhone database. Still gotta work out nicely paginated printing, and to be honest I’m not sure it’s necessary. Will probably be left out on initial release, as anybody can export to CSV and print that, and stick it in a safe.
Once you’re in, you’ll see the main utility window:
One of the key features here, Import, can be fired off pretty easily to import Strip CSV data (more on that in a later post, I have an EBNF that I need to clean up for programmers):
Then, you fire up the iPhone version, select the Sync tab, browse for your desktop on the local network, and begin the sync operation:
Those other buttons on the select switch indicate performing a restore from your desktop, and performing an authoritative override of what’s on the desktop. Once we’ve chosen the desktop we wish to sync with, we go for it (click to embiggen):
Here we can see that a new category called “trekkers” has been added:
And we can also see that the data import supports multiple values for field types on an entry:
The reason this worked is because they were delimited by a | character in the CSV data shown above. It can be escaped with a backspace. Like I said above, more on the import format later. Obviously, my Trek knowledge above is bogus, just needed some data.
Posted: Mar 19, 2010
by Billy Gray
Tagged ruby, rubynation, bestpractices
I’ll be attending this year’s Ruby Nation conference in Reston, VA April 9-10, and I’m quite looking forward to it. The list of speakers and talks is absolutely fantastic, and includes the excellent Nick Sieger, whom I had the pleasure of first meeting at Ruby Fringe in 2008. There are still a couple of open registration seats left if you haven’t signed up yet. If any of the Ruby heads out there want to meet up, I’ll be in town from the evening of April 8th to the morning of April 11th. You can find me on Twitter as @billymeltdown.
I’ve been spending a little time revisiting some of the core concepts and industry practices with Ruby, and recently Bret Morgan of DBL Systems pointed me in the direction of Gregory Brown’s newly published book, Ruby Best Practices. The first chapter alone is excellent, a kind of sermon for Test-Driven Development, and I highly recommend it.
Posted: Mar 18, 2010
by Billy Gray
Tagged pingme, software, reminders
We have some sad news to deliver to users of the PingMe reminder service. Starting today, we will be blocking new registrations on PingMe, and we have decided to turn off the system permanently starting on April 23rd.
So long, and thanks for all the fish
We believe in working on projects that are sustainable. When we built PingMe, we had very high hopes for it. It was designed from the ground up as a system that was more dynamic and flexible than anything else out there.
We have maintained PingMe for free for as long as possible for all the people that have been depending on it. We love to get email from people who have found the service to be an enormous help in their lives, and it’s partly a sense of responsibility to them that has helped us to keep it up.
Unfortunately, it’s just not a sustainable service in the long-term for several reasons:
- PingMe requires continuous maintenance and constant vigilance against abuse by spammers, malicious users, and even inadvertent mistakes. Recently, direct and accidental system abuse cases have been on the rise, to the point that we have almost daily incidents to deal with.
- PingMe is a difficult system to support, because of various problems with external SMS gateway providers, cellphone carriers, and email systems. These issues are mostly out of our control, and make it difficult or impossible to troubleshoot and resolve even common problems.
- Despite the fact that PingMe is widely used and well liked, it hasn’t grown to the point where we can monetize it or effectively compete with other systems.
- We’re a small team, and maintaining PingMe has become an undue burden on other projects that we have brought to market
Switching to Another System
There are plenty of other reminder systems and task managers out there. Each has their own way of doing things. While we can’t recommend any particular system, we encourage you to try out a few and see what else you like.
In order to help you get your data out of PingMe, we’ve added an export feature so you download an Excel/CSV or XML-formatted export of your reminders. We’ve also posted a reference guide to the export data here on our blog.
What does this mean for Zetetic and your other products?
PingMe is the only system we are shutting down. Tempo Time Tracking, Strip Password Manager, Codebook Secure Notepad, etc, will be completely unaffected. We are doing very well as a company, continuing to grow, and approaching our 6th year in business. In fact, the most important reason we are turning off PingMe is so that we can continue to give proper focus to our other products and services.
Thank you
Finally, we want to say thanks to all of our PingMe users for trying the service, providing feedback along the way, and helping to spread the word. We really appreciate it and sincerely apologize for any inconvenience this may cause.
Posted: Mar 18, 2010
by Billy Gray
Tagged pingme, xml, reminders, data, export, csv
Last night we made a small update to the PingMe reminder service to make it possible for you to export a listing of your reminders, should you decide you’d like to switch to another service. If you click on the Profile Tab, you will see an “Export” section at the top of the page:
The export options are straightforward, but I want to describe the CSV and XML formatted output and describe what exactly each of the fields describes, to help anybody looking to massage the data for another system.
CSV Output
CSV (comma separated value) output is suitable for opening in a Spreadsheet program like excel, but it can also be opened with any standard text reader. Field data will be correctly escaped when necessary according to the CSV spec (for instance, when a field contains a comma, line-break, or quote character). The data is UTF8-encoded (not ASCII), but if you try to open the file directly in Microsoft Excel, it may munge the UTF8 characters that extend beyond the ASCII set. Other spreadsheet programs should be able to handle this for you, and help you convert into an actual Excel workbook.
Here’s some example output, with a header row:
id,is_done,require_confirmation,created_at,updated_at,is_active,is_recurring,message,start_date,schedule_type,recurring_frequency,is_pestering,targets,pester_interval,tags,users
172,false,false,22-Jul-2008 17:34,22-Jul-2008 17:34,true,false,check check,22-Jul-2008 17:45,one-time,daily,false,Gmail,0,"",Billy
XML Output
This will produce an valid UTF8-encoded XML document. The root element is <pings type="array" />, and it contains, you guessed it, a list of pings as XML! Type-hinting attributes are turned on, but should be easy to ignore. This is what it generally looks like:
<pings type="array">
<ping>
<created-at type="datetime">2008-07-22T17:34:04Z</created-at>
<id type="integer">172</id>
<is-active type="boolean">true</is-active>
<is-done type="boolean">false</is-done>
<is-recurring type="boolean">false</is-recurring>
<message>check check</message>
<require-confirmation type="boolean">false</require-confirmation>
<updated-at type="datetime">2008-07-22T17:34:10Z</updated-at>
<tag-s></tag-s>
<users-s>Billy</users-s>
<targets-s>Gmail</targets-s>
<kind>one-time</kind>
<recurring-frequency>daily</recurring-frequency>
<is-pestering type="boolean">false</is-pestering>
<pester-interval type="integer">0</pester-interval>
<start-date type="datetime">2008-07-22T17:45:00Z</start-date>
<schedule-type>one-time</schedule-type>
</ping>
</pings>
Export Field Guide
created_at – Creation timestamp. Excel format used in CSV output. All timestamps are in UTC.
id – The unique integer identifier of the ping. There are many others like it, this one is yours!
is_active – Boolean determining whether the ping is still “active” in the system. This is only really set to false by the system, not the end-user.
is_done – Whether the ping has been marked by the user as “done”. You can interpret is_active = false and is_done = true to mean the same thing.
is_pestering – Boolean indicating whether or not the reminder is supposed to bug you at a particular interval until you respond.
is_recurring – Determines whether the ping will be rescheduled after it sent at it’s start_date.
message – The contents of the reminder itself
pester-interval – The interval in seconds that a ping should be re-sent until a response is received from the user. Only valid if is_pestering == true.
recurring-frequency – The schedule to be used when is_recurring == true. Can be:
- daily
- weekly
- monthly
- yearly
- biweekly
- mon-fri
- quarterly
require_confirmation – Boolean setting that, when true, makes the ping a kind of “to-do.” Normally, if a ping has been sent at it’s scheduled time, and it’s not up for rescheduling, it is marked as “done” by the system. However, if require_confirmation is set to true by the user, the ping will not be marked “done” until the user replies to confirm that the task has been completed. This causes the ping to stay on your homepage on PingMe, instead of being archived.
schedule_type – Convenience text describing what kind of ping this is: pestering, repeating, or one-time. Pings that are combined pester-repeat pings are simply marked as “pestering”, so you may wish to consult is_pestering and is_recurring directly.
start_date – The date at which the ping should be scheduled for delivery.
tag_s (tags in CSV output) – A comma-separated list of tags assigned to this ping.
targets_s (targets in CSV output) – A comma-separated list of the targets assigned to this ping (names only).
updated_at – Timestamp indicating when this ping was last modified, by the user or the system scheduler.
users_s (users in CSV output) – A comma-separated list of the users this ping was assigned to, listed by login name. This exists because pings can be shared with more than one user.
Posted: Mar 17, 2010
by Billy Gray
Tagged iphone, encryption, codebook, sqlcipher, secure, notebook, products
Time for a little business, friends! We have a new application in the iTunes App Store, and we call it Codebook: A Secure iPhone Notebook for People with Secrets!
Download it now in the iTunes App Store for only $2.99.
My bias being what it is, I recommend that you replace your other notes apps with this elegantly simple notebook that keeps all your data private, and just gets out of the way. Codebook is perfect for attorneys, physicians, journalists — anybody who needs to keep their notes confidential.
Codebook is built on the same rock-solid encryption engine we use for STRIP, SQLCipher. It’s perfect for storing sensitive meeting notes, terrible poetry, anything you want to keep secret should you lose your iPhone or iPod Touch. ALL data stored in Codebook is stored using the robust and peer-reviewed 256-bit AES encryption implementation in OpenSSL.
I used to use Apple’s included Notes application a lot myself, but I’m getting less and less comfortable storing stuff in there. Here’s my use-case, since I’ve been using Codebook for a few months now:
I, for one, have a penchant for thinking up lots of bad “lyrics” for my band. I’ll be on the subway reading some radical article in Harper’s, and BOOM, I get this great idea! I fire up Codebook, tap in my password real quick, and tap out my idea. If I want to add it to a previous bit of my wondrous witticisms, they’re incredibly easy to find via the Search feature, or just by going back in time and scanning the titles.
It may not be totally evident in the screens here, but we opted to use the first line of each note as its title (an idea we got from John Gruber’s writings about the untitled document syndrome, and the Simplenote app), so you don’t have to think of one. It’s just there, it just works.
One last thing we made sure we included: Email forwarding of individual notes. You just might want to mail something out and we wanted to make that easy to do.
Stephen and I started work on this back in February of 2009, and have basically sat on it since while we put priority on Strip. Recently, I took some time to clean it up and get it ready for a proper release. Feels good to finally have it out there in the store. If you give Codebook a spin, let us know what you think!
