Re: Proposal for file attachment API

[Luke <..hidden..>]

On Tue, 5 Jul 2011, Chris Travers wrote:

> I guess there is one thing that's bothering me about this discussion,
The thing that bothers me (slightly) is that you and I seem to be the only 
people with opinions on this.
> and it's worth bringing up.  I am not aware of a single filesystem
> which attempts to enforce uniqueness on file data.  I would think if
> it was a significant problem, it would have been tackled there first.
My reasoning was this:

If content is not unique, or at least probably unique, you will 
probably end up with many copies of a single document being attached, to 
for example, a quote, an order, an invoice, possibly a purchase order, and 
maybe a payment (just for a most extended case).
I thought that was bad, initially, from a data storage prospective. 
Storing files in a database used to be thought a generally bad idea, but 
if you're going to do it, it seems likely to be a good thing if you don't 
use the storage wastefully: it will have performance effects, possibly 
might have effects when doing cleanup/data recovery/repairs, etc..
The second reason for jumping to uniqueness of contents under the multiple 
links system, was also in part so that a user could upload a file without 
knowing if it was already on-system.  Instead of being stored, a link 
would be created, and the upload thrown away, if the contents were already 
in the repository.
As one, I can attest that users are lazy.  It may be easier to re-upload 
ten times, than to go hunting the already uploaded copy ten times.
That doesn't mean the software should have to maintain ten copies.

The third reason why I don't like multiple copies of the same document, 
although this is probably more of an argument for the linking system, is 
the case of contract law.
Hypothetical:
Manufacturing makes an agreement with a customer, and attaches the 
contract to a quote.
They email it to Joe in accounting for his approval.
He suggests some changes by altering the document and sending it back, but 
they tell him to go ahead as-is.
He puts through an order and invoice to the customer, attaching the file.
Only what he attaches, is his munged version, accidentally.
That'll probably become the legally binding version of the agreement.

I imagine iterations on that kind of case, and my thought for how to 
limit it in software, is to let the order Joe creates, link back to the 
original contract, attached to the quote.
That in and of itself does not require unique contents, but it does 
require the linking scheme.  If you're going to do the linking scheme, it 
seems a small step to make it global, which leads to probably unique 
contents.
(I suppose if you really want to limit that case, we should have file 
name uniqueness at the customer or vendor level, but...oy.)
But, let's consider no unique contents, and no linking.

You may end up with at least three copies of each file on the 
system.  With lots of files, the storage requirements for that are going 
to get absurd.
I, at least, run LSMB in virtual machines.  I don't always grant them a 
huge amount of space, or a huge amount of memory.
I conserve where I can, and if I don't have to duplicate every 1-5 meg 
proposal or file, 2-3 times per customer, I don't want to.
Consider a company which attaches a standard contract to every order, or a 
standard SomethingOrOther.
This probably shouldn't be used for that, but there's a good chance it 
will be.
(Realistically, if that was the system in use, I would avoid passing 
the attachment along the accounting chain like that--I'd put it in the 
first document (quote or order), and refer back to it when I had to.  But 
I'm not everyone.)
N.B. File systems do not require this kind of uniqueness, but the ones 
which assume a level of intelligence in their users, do make it possible, 
via various kinds of links.
If you're really very uncomfortable with it, I'm certainly not going to 
insist upon it, but I do think it makes for a better system in the long 
run, if we try to minimize the number of copies of files as much as 
possible, but maximize the number of documents they can be attached to.
It's the virtual names (I.E. multiple linking) I most wanted, and the 
uniqueness of contents was a by-product idea that seemed good in 
retrospect from a storage prospective.
 I do like your source document reference plan though.

I suppose I am viewing files as their own documents in all this, 
attachable to anything that supports it.
> wondering if the relational model is really well suited for this
To my mind, it's the only model that's perfect for it, perhaps sans the 
primary key issue.
The first virtual FS I ever worked with, was a PostGreSQL backed one, 
although it used non-DB storage.
In it, any number of paths and names could point to each file.

> problem.  After all we are talking about a huge natural primary key so
> however we go about enforcing that, there will be a substantial
> performance cost.
I really wish we could find a way not to use that primary key, or to 
derive a unique short form, so we don't have that problem.
You have a point about checksums, but there aught to be a way to 
fingerprint a file and do comparisons on that basis.
> > I agree with that, but if we have to...  Do we have other tables in 1.3
> > where that's the case?
> Yes.
>
> note (abstract table, no acfcess)
> entity_note (notes for companies and people)
> eca_note (notes for customer/vendor credit agreements).
>
> All could conceivably be queried together with:
> select * from note, but to insert you must insert into the appropriate
> subclassed table.
>
> There is no attempt to enforce uniqueness of note contents.
Haha!

(Although, if notes were likely to be several hundred K to a few meg each, 
someone would probably suggest it.:))
Luke