For the last two years, I have used a combination of calibre and Zotero to manage my research sources, mostly PDFs of journal articles. Zotero is great for reference management and automatic citation. I force my first-year seminar students to use it for their research papers, and even upper division students who have not used a reference manager before have their minds blown when they realize they no longer need to keep track of where the commas go in a citation. Enough people love Zotero that I hardly need to sing its praises. But I don’t like its obtuse, iTunes-like management of your library in a hierarchy of meaninglessly named subfolders, and I also wanted something that could sync my library to my Kindle DX.

I started using calibre, which is primarily intended as an e-reader manager, for that reason, and also for the fact that it allows easy conversion to e-reader file formats from HTML, DOC(X), and also from EPUB (which the Kindle can’t read) to MOBI (which it can). (It does offer PDF conversion as well, but I found the results to be not very good, so I read native PDFs on the Kindle.) Additionally, while calibre also does not give you control over the library folder hierarchy and file naming, it at least uses a human-readable hierarchy based on <author(s)>/<article name>/<article name>.<extension>. (Article name is duplicated in the folder hierarchy and the file name because multiple formats of the article, such as PDF and EPUB, may coexist in the folder.)  This meant that when I found a journal article I wanted to read, I had to import it to calibre, import the bibliographic data to Zotero, and link from the Zotero database to the file in the calibre storage hierachy. While no single step is difficult in and of itself, doing three things instead of one always inspires a certain amount of laziness, so it meant that my libraries often got out of sync. In practice this usually meant that I had the article to read on my Kindle, but not in Zotero. Additionally, time I spent updating metadata in calibre had limited payoff because I couldn’t use calibre for citation.

Since nothing sidetracks writing a paper like finding out your sources aren’t in order, I’ve decided it’s more important to prioritize the reference manager, and want to use Zotero to manage my library. Fortunately, Zotero has some ways to add these desired features, including the ZotFile plugin for library management (which has been around for awhile, but since I was managing my library in calibre, I haven’t used ZotFile until now). ZotFile lets you specify an alternate location to store your attached files, and gives you control over the folder hierarchy and file naming (similar to Mendeley, or to many music players; note, calibre also gives you this kind of control, but you have to export your library to an alternate location, duplicating all your files–it doesn’t allow you to mess with its internal library structure). The instructions for doing this on the developer’s web page were a little unclear to me, so I went looking for help in the Zotero forums, and found a thread from today started by a new user who was looking for guidance in how to do the same thing. In this post, I’ll explain how to accomplish this.

First, you have to understand that Zotero draws a distinction between saving a file internally (which it calls “stored” files), and saving a link to a file anywhere on your hard drive, though both stored and linked files are called “attachments”. Part of the reason why its internal naming is so obtuse is probably because they don’t want you messing with it and moving files around. If you want a pretty, human-readable hierarchy, you need to tell ZotFile how to construct that hierarchy. Since linked files can be anywhere on your hard drive, by default Zotero saves their location using absolute paths. If you want to share your files across computers, you need to instead tell Zotero to interpret all file paths relative to your human-readable hierarchy.

1. Go to Zotero Preferences (using File→Preferences, or using the gear icon). Click the Advanced icon at the top, then the Files and Folders tab.
2. Under Linked Attachment Base Directory, hit the Choose button and use the file browser to pick the folder where you want your library to be organized:
   NOTE: You do not have to change the Data Directory Location. This is the location of your Zotero database, and also where PDFs are stored internally, but the whole point of what we are doing is to not store the files internally.
3. Now go to ZotFile (not Zotero) Preferences, by clicking the gear icon, and go to General Settings.
4. Under Location of Files, click Custom Location, and set it to the same folder you just set as the Linked Attachment Base Directory.

   

5. Check “Use subfolder defined by”. The default, `/%w/%y`, will organize your files into directories by journal/publisher (uses the publisher name if there is no journal name, e.g. for books) and year. Since I want to be able to navigate this directory sensibly, and I can’t always remember the year a paper was written, I dropped year from the definition. The codes used here and in the next stop are listed in detail at the ZotFile website.
6. Click the Renaming Rules tab. Leave “Use Zotero to rename” unchecked. The default naming rule is `{%a_}{%y_}{%t}` which would generate something like “Smith_2013_This is my title.pdf”. I don’t like underscores in my filenames, so I set my renaming rule to `{%a }{%y – }{%t}`, which would produce “Smith 2013 – This is my title.pdf”, and changed the delimiter between multiple authors to an ampersand surrounded by spaces. Others might prefer a comma. If the filesystem can handle long names, I don’t see why I shouldn’t take advantage of that, so I unchecked both “Truncate title after . or : or ?” and “Maximum length of title”. I left the maximum number of authors at 2, and the use of “et al” for more than two authors.
7. At this point you can select an item in the main Zotero window and click the Update button at the top of the dialog to see how the file will be renamed. If you are satisfied with it, hit the Close button.
8. Now we want to actually move our attached files to this new hierarchy. I began by sorting my library on the attachments column (just click on the paper clip at the top of the column). Then Shift+Click to select all the items with attachments. Right-click on the selected files and choose Manage Attachments→Rename Attachments. ZotFile will now move the files to the Location of Files specified in the preferences. It will do this for both attachments, like the files in my calibre library, and Zotero’s own internally stored files, which will become attachments in the new location, and no longer be stored internally. But this will not move web page snapshots, which stay internal.

Do keep in mind that ZotFile does not copy the files, it moves them. Since my attached files were in my calibre library, and I’m not giving up on calibre just yet, I didn’t really want them deleted from their original location. I backed up my calibre library so that I could allow ZotFile to move the files to the new location, then restored my calibre library from the backup. But I kept the backup for the next step. Since I had a lot of sources in the Zotero database (hundreds) that were not linked to the PDFs in calibre, I could link to the PDFs in this backup, then move them with Manage Attachments→Rename Attachments without affecting my calibre library.

Storing the files this way has two advantages over storing them internally. First, since they are in a human-readable hierarchy, I can pretty quickly find a file through the file browser of another application—for example, if I am writing an email to a colleague and want to attach an article— without having to switch to Zotero first. Second, although Zotero supports WebDAV for library syncing, and Zotero’s own storage is not that expensive, using ZotFile to organize your attachments this way allows you to use any cloud service to keep your library synced across computers, such as Dropbox (mentioned by the new user who started the forum thread linked above) or SpiderOak (my preferred backup/share/sync service).

I ran into a little bit of an issue in that I use Zotero Standalone. This requires me to also have Zotero for Firefox installed so that I can import sources while web browsing. But the preferences did not propagate between them (for either Zotero or ZotFile), so I had to independently set the Linked Attachment Base Directory, Location of Files, and Renaming Rules in both Standalone and the Firefox plugin.

While I’m pretty happy with the result, this all leaves calibre and Kindle syncing out of the equation. Using external PDF reader applications, ZotFile can also manage syncing PDFs, including annotations, between your Zotero library and iPad or Android tablets. Maybe there’s a way to get this to work with calibre to keep my library synced with my Kindle? That will be the next thing to figure out…

Thanks to Zotero forum users bwiernik and adamsmith.

Last weekend I saw an episode of The Good Wife (it’s a few weeks old, I’m a little behind) in which the governor-elect of Illinois decides to send a message to fictional tech giant ChumHum by publicly floating the possibility of collecting taxes on the sales of out-of-state, internet-based companies. Then at the beginning of this week the Supreme Court declined to hear Amazon’s appeal of New York State’s real-life law requiring just that. (ChumHum always struck me as a Google stand-in rather than an Amazon stand-in, but, whatever.) People tend to view Amazon as the evil giant in this case, with some commenting that “It’s about time” they started paying their fair share. But I think there’s a tendency to lump this in with more general concerns about corporate tax avoidance, and I want to suggest some reasons why the new regime is undesirable.

First, Amazon isn’t necessarily going to be the one paying the tax. Sales taxes tend to get split between the seller and the consumer, with consumers absorbing more of the tax on necessities (it’s difficult to stop buying food, or, for that matter cigarettes), and sellers absorbing more of the tax on luxuries. Now, Amazon is probably mostly selling things that fall more toward the luxury end of the spectrum, but some of the tax will be absorbed by consumers. Of course, the real point of the law is that requiring Amazon to collect sales taxes levels the playing field between out-of-state and in-state businesses, but some sales Amazon would have made won’t be substituted by an in-state purchase, because the consumer will decide that the after-tax price is too high. This represents a loss to Amazon, no gain to any in-state seller, and a loss to the consumer. Some consumers will still make purchases, so the state government will still collect new revenue.

Second, and more important, sales taxes are fairly regressive. Lower-income households spend a relatively greater proportion of their income than upper-income households, so if we consider the tax as a percentage of income, lower-income households are paying a higher percentage. New York tries to make the sales tax less regressive by exempting necessities like food, but it’s tax exemptions are notoriously bizarre. For example, nuts are not taxable, unless they are honey-roasted, in which case they become taxable again. Rather than extend a regressive tax to out-of-state purchases, New York could reduce the sales tax on in-state purchases. The lost revenue could be made up by increases in the income tax or (for localities like New York City that rely on a sales tax increment) the property tax. This would be a more progressive outcome, but of course it is much harder politically to raise taxes on in-state sources than to impose a tax that seems to fall on an out-of-state source. But since the sales tax reduction would increase in-state sales, in theory the lost revenue could be offset (this would need to be modelled) by increased revenue from business income taxes and from property taxes, even without an increase in those tax rates.

Third, the whole reason this is an issue is because it was relatively easy for consumers to avoid the sales tax by ordering from out of state. One could argue that governments should try to avoid taxes which are easily avoided. Out of the big three, sales taxes are most easily avoided, income taxes less so (there is some debate over the impact of tax-related migration, but one recent study flat out calls tax flight a “myth”, Gerard Depardieu’s move to Russia notwithstanding), and property taxes are least easy to avoid. In fact, the land portion of the property tax is completely unavoidable, while the capital (structure or building) portion may cause some capital flight if the tax is much higher than in competing jurisdictions. Because sales taxes are relatively easy to avoid, and because there are a large number of taxing jurisdictions that businesses have to keep track of, there have been some noises in Congress about standardizing sales tax rules across the states. This is a bad idea for reason number two, regressivity. Consumption taxes are pretty popular among conservatives as an alternative to the income taxes, I would argue specifically because consumption taxes are so regressive (but see Robert Frank’s discussion of a progressive consumption tax). My concern is that if we back ourselves into a national sales tax, it will gradually eat away at the income tax, leading to an overall more regressive tax system.

Finally, to the extent that this tax does fall on Amazon, we can ask what kinds of businesses we want to be taxing. The Mining, Quarrying, and Oil and Gas Extraction sector pays on average 6% of corporate income in taxes [gated link]. So if we were looking for new revenue…

The Bronx is heavily Black and Latino, and shows a high number of NYPD stops over much of the borough.

There have been a lot of ups and downs in the battle over the NYPD Stop and Frisk program. Over the Summer, a federal judge ruled the program unconstitutional as practiced, and appointed a federal monitor to oversee reforms. Then, a month ago, a federal appeals panel removed Judge Scheindlin from the case, and put a stay on her orders. Bill de Blasio was  elected mayor promising to drop the City’s appeal, but lame duck Mayor Bloomberg tried to push the appeal to a full overturning of Judge Scheindlin’s ruling before leaving office. Now the full Second Circuit has put on hold any further action while the litigants negotiate—clearly taking the ball from Bloomberg and passing it to the mayor-elect.

Stops are concentrated in predominantly Black and Latino neighborhoods like Bedford-Stuyvesant and East New York in Brooklyn, and East Elmhurst and Jamaica in Queens.

Much of the focus in the case has been on whether the Stop and Frisk program was implemented in a racially biased manner, with considerable concern over the targeting of young Blacks and Latinos. The maps I’ve created, based on the 2011 statistics, do show many more stops taking place in predominantly Black and Latino neighborhoods. The maps do not show the race of the people being stopped, an important consideration in the trial, as plaintiffs presented evidence that the “hit rate”—stops that actually led to the discovery of a crime—was much lower for Black and Latinos than for Whites, indicating that the police were using a different, and looser, standard for what they regarded as suspicious behavior by Blacks and Latinos.

The main exception to the  correlation between neighborhood demographics and volume of police stops is in Manhattan. We do see a concentration of stops in the Black and Latino neighborhoods of Harlem and Washington Heights, but we also see generally higher stops in Manhattan than in the outer boroughs. Doubtless this is due to Manhattan’s generally higher nonresidential population (commuters, shoppers, etc.), as well as the generally higher pedestrian counts in Lower Manhattan and Midtown.

Manhattan, with its high daily influx of nonresidents for employment, shopping, and commercial activities, has high numbers of NYPD stops throughout the borough.

While the differences in “hit rate” helped Judge Scheindlin conclude that this policy led to “indirect racial profiling”, the ridiculously low yield—less than 12% of stops led to a summons or arrest in 2011 (about equally split)—suggests that the police are extremely poor judges of suspicious behavior. But what do you expect of a policy that allows the police to stop someone for “inappropriate attire for season” (about 7.5% of stops in 2011)?

Staten Island’s population is more White, and subject to fewer police stops.

The tagline of this blog is “Urban Economic Geography and Open Everything”. Free City supports free and open source software, open access publishing, and open data. Many governments are making much of their data publicly available, including New York City’s own open data portal. But geospatial data has sometimes been handled differently, sometimes because of privacy concerns, often because of security concerns. For example, the New York State GIS Clearinghouse allows direct downloading of orthoimagery (aerial photography) for much of the state, but for sensitive areas, including all of New York City, there is are some hoops to jump through.

Doubtless some of the evocation of privacy and security concerns is disingenuous, as much of the protected data is nonetheless commercially available. But in some cases it is the government itself which is licensing the data, raising the question of why they are only willing to give it to people who pay for it. In two recent cases having to do with property records, local governments chose or were forced to make their geographic data freely available. New York City’s Department of City Planning recently chose to make its MapPLUTO database available for download, whereas previously they had charged $300 per borough for this collection of building and tax records joined to parcel footprints. On the other side of the country, Orange County, California was forced by a court decision to open up their parcel database, which they had previously been licensing for $375,000!

In the Orange County case, the county was forced to release the data under the existing Public Records Act. Why wasn’t it already available? Because it was GIS data! The county claimed that the data was exempt because it was “software”. The California Supreme Court sensibly rejected that claim. After all, if the same information had been requested in hard copy, the county would have been legally required to provide it (and would have realized as much). I would go even further and ask, why is there an exemption for software? As a free software advocate, I would argue that the same reasons that apply to public records also apply to publicly developed software, including that it is paid for with tax dollars, and that there is a public benefit to its wide availability. There could still, as is the case with public records, be security exceptions for certain kinds of software.

The lawsuit which led to this case was filed by the Sierra Club, which had requested the parcel data for use in conservation planning. As a nonprofit, they could not afford the $375,000 licensing fee. Now that the court has ruled that the data are not exempt from the existing law, the Sierra Club will be able to obtain the data “for the cost of producing the physical copy”, presumably the cost of some DVDs, or free if provided electronically. In the New York case, the fees weren’t nearly so egregious, but nonetheless, I knew of small nonprofits, and—until the Grad Center licensed the data—CUNY grad students who could not afford to pay them.

The size of the fees—a lot for a grad student, not a lot for a government the size of New York City’s—really call into question why they existed in the first place. At $375,000 a pop, Orange County claimed that the fees helped support the GIS services that made the data available in the first place. But NYC’s Department of City Planning was only collecting $50 to $80 thousand a year from their licensing scheme, which is pretty small potatoes. Furthermore, at least one nonprofit obtained the data for free through a NYS Freedom of Information Law Request! So DCP, unlike Orange County, did not even attempt to argue that the data were exempt from its state’s public records law. But they did require the nonprofit to sign the standard license agreement, which restricted redistribution of the data. This does seem to suggest that, rather than pecuniary interest, DCP was more interested in asserting control over the data for some reason or another, and that there was something about this data in particular that it was concerned about, since it provided virtually all of its other data for free via public download.

Thanks are due to Steve Romalewski, my friend and former boss at Center for Urban Research. Steve has advocated for the freeing of this data set for some time, and in announcing the new DCP policy on his blog, he thanks several others in academia, journalism, and the nonprofit world for their contribution to this wonderful result.

Mayor Bloomberg’s proposed ban of sugary drinks larger than 16 ounces has been making national news. Industry groups are lined up against it, and 64% of New Yorkers are opposed to it. The industry is organizing opposition around the concept of “freedom of choice”. Jon Stewart has repeatedly made fun of the fact that in New York, carrying 25 grams of marijuana is now “legal” (actually, a violation which can be ticketed but which one cannot be arrested for), while drinking a large soda is “illegal”.

It of course would be silly (so I will do it) to criticize a humorist on facts, but pot smokers (or carriers) will still be issued citations, but can no longer be arrested. This change was necessary because the police, under the controversial Stop-and-Frisk program, were regularly stopping young black and Latino men, asking them to turn out their pockets, and then arresting them for “displaying” marijuana, when merely “carrying” marijuana was a violation. Now displaying or carrying are treated the same. This is not  a major shift in drug policy, and to my mind doesn’t do nearly enough to rein in the utterly offensive Stop-and-Frisk program.

Soda carriers, or displayers, won’t be subject to ticketing or arrest under the proposed “ban”, but certain businesses won’t be allowed to sell large sodas. As far as I can tell (I couldn’t find an “official” proposal—if you can find one, please let me know in the comments) this would only apply to food service establishments. You could still buy a two-liter bottle of Coke at the corner bodega and walk down the street drinking it. Meanwhile, let’s compare the damage done by soda versus the damage done by marijuana. Estimates of obesity-related mortality are in the hundreds of thousands annually. Estimated deaths due to marijuana use are approximately zero, give or take nothing. (In fact, a Google search of “marijuana mortality” turned up this study indicating that marijuana use may lower mortality among people suffering from psychotic disorders.)

But Stewart’s comparison of food policy and drug policy is apt. Certain foods (sugary drinks, but not broccoli) and certain drugs (methamphetamine, but not marijuana) lead to extremely serious health problems for some people. And many users, of sugar and of meth, want to use less, but have a hard time stopping. One message of the poll linked above is that almost 3 million New Yorkers (36%) approve of the proposed ban. How many of them are people who want to reduce their caloric intake, but find it hard to avoid when every fast food joint tries to upsell them to the larger cup size? It is well-established at this point that (a) behavior is strongly influenced by the social environment, and (b) short-term gratification usually trumps long-term rationality, even in people who want to make and stick to the long-term rational decision. So freedom of choice should include the freedom to drink soda, but also the freedom to live someplace where one isn’t continually encouraged to engage in unhealthy behaviors.

Robert Nozick, philosophical champion of libertarianism, concludes his influential Anarchy, State and Utopia with a chapter entitled “A Framework for Utopia”. After spending a book defending the idea that there is no justification for a more-than-minimalist state—i.e., no paternalistic prohibitions, no economic redistribution—he argues that this only applies to a national or central state. Local communities, from which people would be allowed to exit at any time, could allow things, such as public nudity, which many or most other communities find offensive; or prohibit things, such as large sugary drinks, that many or most other communities find unexceptionable.

The main complaint that can be leveled against Mayor Bloomberg’s plan is not that it isn’t good policy, but that it is policy not democratically arrived at. If people should be free to live someplace that prevents them from drinking too much soda, they should also be free to live someplace where they can drink as much soda as they want. The questions become (a) at what scale should the rule be implemented, and (b) how do you deal with the transition? Clearly these kind of rules should be applied at the local level, not the federal level, and probably not at the state level either. Nozick argued these kind of restrictions should only be allowed in “face to face” communities. I take that to mean something approximating a neighborhood, though I think in order to be effective that many policies (particularly economic and planning policies) would have to be implemented at something like a metropolitan area. New York City is a unique case, as it is larger by itself than many metropolitan areas, but still only one part of its own metropolitan area. Thus a policy which would be thoroughly justifiable if enacted by a small suburban municipality, is more ambiguous in the case of New York.

Here Mayor Bloomberg would be well-advised to consider the opinion of New Yorkers, which appears to be against the plan. But if people should be free to choose to live in a place with a soda ban (or a drug ban, or a cheese ban), what do you do when a substantial minority wants such a policy, but no single place has a large enough majority to enact it? That is, what if 34% of the nation wanted a local soda ban in their community, but no single municipality had a majority in favor of the ban? In this case, it might be allowable for some local government to enact the ban in spite of local opposition, and expect population to gradually equilibrate to the new reality—as some residents found that they did indeed like the ban in spite of their earlier opposition, and others moved away and were replaced by new residents who moved in with full knowledge of the rule. Again, New York City’s unique size in the American system of cities makes the justifiability more ambiguous. A better approach for Mayor Bloomberg might be to implement the policy only in certain neighborhoods, with opportunity for public participation in choosing which neighborhoods would be affected.

Open Access publishing refers to making scholarship—primarily journal articles, but data is often included here as well—freely available to the public. Open Access became international news earlier this year when mathematician Timothy Gowers announced that he was boycotting Elsevier, a major academic publisher, and thousands of other researchers signed (and are still signing) on. (For a short overview of Open Access, see the just-published Open Access and the Future of Academic Scholarship by Barbara Fister—thanks to Maura Smale @msmale for the link.) Continue Reading »

PostGIS and SpatiaLite offer a large number of spatial query operations, including buffers, intersections, and spatial joins. It’s really useful, especially when you’re still experimenting, to visualize your results without having to create a new spatial table, spatial view, or exporting to a shapefile or other GIS data format. Fortunately, there are several ways to do this, including at least three plugins for Quantum GIS: Fast SQL Layer, PostGIS Query Editor, and RT SQL Layer. In this blog post I will walk you through using Fast SQL Layer. Continue Reading »

Normally I don’t write about hardware on this blog (or pretty much anywhere), but I’m making an exception in this case to shout out a thank you to Theje for a YouTube video showing how to fix a weird keyboard problem on my IdeaPad Y560p. Certain keys quite suddenly started doing strange things: The H key would act like an up arrow, so while you were typing, your words would suddenly start appearing in the middle of the last line. Continue Reading »

Colleagues at Center for Urban Research have posted a fun interactive map for visualizing the various New York State redistricting proposals. New York, like all states, is redrawing state and federal legislative districts based on the 2010 Census. So far, it’s not going so good. The proposals released by the legislature have been widely criticized for not preserving “communities of interest”, a common redistricting desideratum. The legislature was unable to agree on federal districts, and left that up to a federal court. State legislative districts were signed into law by Gov. Cuomo, in exchange for a legislative pledge to create a bipartisan redistricting panel for the next go round (i.e., 2020). But the redistricting reform that is shaping up has been criticized by a former New Yorker and current member of the California Citizens Redistricting Commission for, among other things, leaving final adoption in the hands of the legislature. Continue Reading »

Why SpatiaLite?

I spent this weekend getting the SpatiaLite GUI installed, with help from SpatiaLite Users Google Group. SpatiaLite is an open source source geodatabase built on top of SQLite, itself an open source database. As a lightweight, “server-less” database, SQLite is frequently used as an embedded database by other applications, including Firefox, Google Chrome, Skype, iTunes, Clementine (my preferred FOSS music player), and Dropbox (on the client side). Continue Reading »