In Judge Peck’s recent Da Silva Moore, et al., v. Publicis Groupe, et al. opinion (2/24/12), he emphasized the importance of having a well-thought out workflow and process when using TAR. In that case, the parties had more than three million emails and they agreed to the use of predictive coding. (Predictive coding is a particular type of search technology that generally relies on Latent Semantic Indexing, a statistics-based search methodology.)

The parties are now in a side dispute over the “scope and implementation” of that use of predictive coding technology in the eDiscovery portion of the case. This highly public and contested argument over which protocol and workflow to use when deploying TAR is the centerpiece of the dispute.

Perfection is never a goal of any technology assisted review software: rather, the goal is to find a reasonable and cost effective way to identify and produce relevant documents while identifying and withholding privileged material. The exact workflow and process has until now been the producing party’s purview. Here, the parties agreed to the defendant’s use of predictive coding, but now disagree on how to use it and the number of documents to be review from a statistically significant standpoint.

In Da Silva, the defendant agreed to turn over the seed set of non-privileged documents used to “train” the system. This disclosure would allow the plaintiff to see the decisions and coding calls made on the seed data. Seven iterative rounds of review were scheduled for the training phase. Once complete, the defendant was to sample the documents not marked relevant by the machine. The sample size the defendant’s were using for this phase was 2,399 documents. The plaintiffs questioned and disagreed that this protocol and workflow would work.

The Court reminded the parties that computer-assisted review “works better than most of the alternatives, if not all of the [present] alternatives. So the idea is not make this perfect, it’s not going to be perfect. The idea is to make it significantly better than the alternatives without nearly as much cost.”

The court agreed to use that workflow as described above, but with the caveat that the parties may need to revisit the proposed workflow if things aren’t stabilized or working as first thought. The fundamental disagreement is over whether the seed set of data used to train the predictive coding technology is representative of the entire data set.

All of these things are mere details of a workflow, which is used to decide how to actually locate and confirm a document is relevant (and not privileged) and marked to be turned over in a document production.

Judge Peck went on to say that the Daubert standard is “not applicable to how documents are searched for and found in discovery.” He also said that “it is the process used and the interaction of man and machine that the courts need to examine.” That interaction is the workflow and process behind what was done with the technology by the humans. That workflow is the cautionary tale from the Da Silva Moore decision. That workflow is what we need to be paying attention to when we are putting together an effective eDiscovery plan.

I consider the next to last sentence of this opinion to be the most use: “As with keywords or any other technological solution to ediscovery, counsel must design an appropriate process, including use of available technology, with appropriate quality control testing, to review and produce relevant ESI while adhering to Rule 1 and Rule 26(b)(2)(C) proportionality.” In the end, workflow matters.

Just last month, in Race Tires, et al v. Hoosier Racing Tire Corp, et al, No. 11-2316, (3^rd. Cir. March 16, 2012), the 3^rd Circuit denied almost all costs relating to ediscovery.  More specifically, it applied a narrow interpretation of § 1920 (4) and held that the plain language of the statute, as applied to ediscovery, only allows reimbursement for scanning and file conversion, i.e., tasks akin to modern day copying.  But nationally, a slight majority of federal district courts have held otherwise.

For those of us in California, several district courts have expressly approved recovery for a broad range of ediscovery tasks.  These have included: collection, processing, and reproduction in preparation for document review.  See Parrish et al v. Manatt, Phelps & Phillips, LLP, et al, No. C 10-03200 WHA (N.D. Cal. April 11, 2011) at 4.  Project management has also been held to be a recoverable cost.  See Jardin v. Datallegro, Inc., et al, 08-CV-1462-IEG WVG (S.D. Cal. October 12, 2011) at 11.

The courts that have addressed these issues generally tend to recognize the utility of ediscovery tasks, i.e., that the underlying tasks were “necessarily obtained for use in the case” rather than for mere convenience—a threshold showing per § 1920 (4).  The Parrish court stated that such costs were “necessary expenditures made for the purpose of advancing the investigation and discovery phases of the action.”  And in Glenn Tibble et al v. Edison International et al, CV 07-5359 SVW (AGRx) (C.D. Cal. August 22, 2011), the court noted that ediscovery costs are necessary because litigants are “required” to produce electronically stored information, unless they can demonstrate undue burden or cost per FRCP 26(b)(2)(B).

Another issue that has come up is “excessiveness”.  In ediscovery, it’s easy for a litigant to say that the opposing party’s ediscovery costs were excessive—ediscovery is expensive generally.  However, being proactive about selecting a vendor can minimize this potential argument.  For instance, in Tibble, the court allowed costs, in part, because the requesting party selected its vendor based on vendor expertise and a competitive bidding process.  (See Tibble at 9.)  (As an aside, the court did not state that the litigant chose—or was required to choose—the lowest bidding vendor.)  As an aside, counsel should be aware that the specific role a vendor plays, and even the language used in invoices, may also impact the recoverability of costs.  This relates to the distinction between attorney’s fees and costs.  If a request for vendor costs appears to describe tasks that involve strategy or other activities “typically entrusted to lawyers”, a court will likely deny cost recovery.  (See Jardin at 11).

In the end, no matter what jurisdiction is involved, to minimize the impact of costs, counsel should consider the full breadth of strategies available, including protective orders and cost shifting.  Further, and perhaps most importantly, even though federal district courts in California have a tendency to allow for the recovery of ediscovery costs, counsel should keep one point in mind as a guiding principle: ediscovery is technical.  Never assume that the court “gets it.”  Counsel should make an effort, at every opportunity, to educate the court and explain the many nuances when dealing with electronic evidence.  It will undoubtedly impact the bottom line.

Parrish: http://law.justia.com/cases/federal/district-courts/california/candce/3:2010cv03200/229842/100/

The parts of the order that concern me the most are:

6. General ESI production requests under Federal Rules of Civil Procedure 34 and 45 shall not include email or other forms of electronic correspondence (collectively “email”). To obtain email parties must propound specific email production requests.

One of the main purposes of the 2006 update to the FRCP was to define ESI (Electronically Stored Information) which includes email. Thereby arguments over email could be eliminated: whether they were about specifically identified eDiscovery requests or whether they should be turned over like any other relevant evidence, regardless of whether it is electronic or not. Yet, the model order requires email to be requested in a specific “email production request.” This seems to be a major step backwards.

5. General ESI production requests under Federal Rules of Civil Procedure 34 and 45 shall not include metadata absent a showing of good cause. However, fields showing the date and time that the document was sent and received, as well as the complete distribution list, shall generally be included in the production.

The model order also calls out metadata specifically (not just that related to email) and says that production requests should not include metadata (unless it is send and receive metadata). This shows a fundamental misunderstanding of ESI, and how it is used in litigation. Metadata exists as part of an electronic document or email. Keeping metadata out of productions is like turning over half a piece of paper without relevant text on it or not turning over the pencil markings on a (paper) memo. Metadata is a crucial part of ESI, even if it isn’t immediately obvious from the face of a document. Moreover, it is the foundation for keyword and other searching techniques.

10. Each requesting party shall limit its email production requests to a total of five custodians per producing party for all such requests…

Limiting an email production request to five custodians demonstrates an arbitrary decision which – more than likely – has little relevance in a case. Every case differs and may involve 5 inventors or a team of 50 inventors. The opposing party generally doesn’t or can’t know who the most relevant and “effective’ custodians without a review of a custodian’s data. Some custodians may have more relevant data than others. Narrowing a search and review strategy to collect and review data from those “priority” custodians is key, and it is key to do this early. Arbitrarily limiting email production requests to 5 people completely misses the point of early case assessment and evaluating the actual data to figure out who are the most relevant custodians.

11. Each requesting party shall limit its email production requests to a total of five search terms per custodian per party… Indiscriminate terms, such as the producing company’s name or its product name, are inappropriate unless combined with narrowing search criteria that sufficiently reduce the risk of overproduction. A conjunctive combination of multiple words or phrases (e.g., “computer” and “system”) narrows the search and shall count as a single search term. A disjunctive combination of multiple words or phrases (e.g., “computer” or “system”) broadens the search, and thus each word or phrase shall count as a separate search term unless they are variants of the same word. Use of narrowing search criteria (e.g., “and,” “but not,” “w/x”) is encouraged to limit the production and shall be considered when determining whether to shift costs for disproportionate discovery…”

Where to start with this provision? There are many disconcerting things here so I will focus on my fundamental concern: the importance of developing a search strategy.  If you have participated in a search strategy session where the objective is to scope a data set for collection or review, there are many, many ways to filter and cull data and there are many, many ways to search for relevant or privileged data. Most of these methods do not involve keyword search AT ALL. The most effective methods for that data set may not even use keyword search at all. I’m not saying that you wouldn’t use keyword search as part of your search strategy, but it certainly isn’t the ONLY method to use. Having a model order suggest a technology and then dictate how to count the searches performed with that technology seems misinformed and fraught with the likelihood of costly motion practice because of discovery abuses.

9. Email production requests shall identify the custodian, search terms, and time frame. The parties shall cooperate to identify the proper custodians, proper search terms and proper timeframe.

What I appreciate about the model order is that parties are required to cooperate with regard to email production requests. The model order misses the point that eDiscovery requires a well thought out search strategy to find the right data, not just run a limited set of 5 keyword searches over a limited set of 5 custodians’ data. A more effective model order would have required cooperation for all ESI requests and not tried to dictate how data should be requested or searched. A search strategy is privileged work product but part of any strategy is cooperation with the requesting party to properly limit the scope of the request and production.

The e-discovery industry ecosystem has some interesting in-built tensions. In some sense, everyone competes with everyone else. Take the following, highly common, situation.  A corporation hires a law firm, and separately an e-discovery service provider, asking the two to work together on a matter (or series of matters).  The law firm has internal capability to handle some, but not all, of the functions that the e-discovery service provider has been contracted to deliver. The software being used through the process is currently licensed by the service provider.  The software parent company is actively selling the software directly to the end corporation (and the law firm for that matter), downplaying (dramatically) the required investment in people, training, and technical support required to make it sing.  The corporation itself is handling part of the process (especially on the front end) and is evaluating its own internal capability and delivery in the e-discovery space. At the outset of the matter, few “lines in the sand” are drawn as to exactly who is going to be doing what.

Sound far-fetched?  It’s more like the norm.  What’s the result?  Everyone looking over their shoulder. Less information sharing than there should be. Excitement at “catching the other party out.” An atmosphere of tension and tentativeness, rather than of passion for excellence and partnership. I talked with someone working for a competitor of ours managing a large-scale litigation. I said, on a scale of 1 to 10, how are you doing?  His answer: “The corporation loves us – it’s a 9.  The law firm?  It’s a 1.”  That can’t be good.  Value is being destroyed in terms of wasted time, additional cost, and unnecessary risk. 

What’s the answer?  We can take some guidelines from game-theory.  I was an economics major.  It’s all a little hazy now, but I remember the Prisoner’s Dilemma http://en.wikipedia.org/wiki/Prisoner’s_dilemma.  The situation demonstrates how people often don’t cooperate, even when cooperation is in their joint best interests. Cooperation becomes more likely with repeated games, rather than one-shot games (so that trust can be built up), and with better signaling of intentions and actions (so parties know what the other is going to do).  Repeat litigation corporations should move away from one-off / fragmented models, and settle on a “repeated game solution” for their e-discovery, with (in my opinion) a single e-discovery provider contracted over a significant period of time.  Similarly, law firms representing the more one-off litigants should do much less provider-switching and much more process integration work with their e-discovery support teams. Finally, in all cases, roles and responsibilities should be clearly delineated – or “signaled.” Make it clear who has ownership for what, and why. Have each player play to their strengths, focusing on where they are world class.  The drive to predictive review may be a fundamental industry shift, but at least as fundamental is the industry’s need, led especially by corporations as the ultimate clients, to drive towards cooperation.

 Reasonableness acts as an undercurrent throughout the civil judicial process but often is pressed to its limits in discovery. Closely related to the concept of “dual proportionality” (more on that in a future post); reasonableness governs our decisions by, ostensibly, comparing our actions and decisions to those of the fictitious “reasonable person”. In practice, however, we often find ourselves checking our decisions against what a judge – and most often a magistrate judge – would consider reasonable in light of the conferences that he has attended and the propaganda that he has heard. The standard varies widely. I rode the elevator the other day with a judge who stared blankly at me when I told him I worked in electronic discovery and asked, without a hint of sarcasm, whether this was a legitimate part of the legal field (a reasonable question if ever I heard one). Compare that to U.S. Magistrate Judge Peck’s, decision in Da Silva Moore, in which he shows a basic understanding of assisted document review technology but displays a deep and inherent understanding of reasonableness in light of the parties’ agreements and the technology available. With a judicial spectrum of understanding that broad, it’s no wonder that litigators often wish that electronic discovery would just go away.

 Ultimately, reasonability is what the Sedona Conference clumsily attempted to address with its Cooperation Proclamation and it is what the Rules Committee intended when it amended FRCP 26(f) et seq., to include – among other things – discussion of the form of document production and issues related to the discovery of electronically stored information. From my own experience, details relating to electronic discovery are rarely ever discussed in anything but a cursory way during the 26(f) conference today, in sharp contrast to 2007 and 2008 after the rule was promulgated.

 We are told repeatedly through law school that “reasonable minds may differ” and it is this point that makes cooperation such a persuasive motivator. If counsel can meet-and-confer in order to agree upon basic, discovery procedures, such agreements are per se reasonable and rarely subject to judicial intervention, much less reversal. This holds true for many phases of discovery, from preservation strategy and collection methodology to search term iteration and review protocols. So back to Sonya’s post from last week: it doesn’t matter which door you choose; in sharing your proposed approach to review (technology-assisted or otherwise) your attempt at cooperation will enhance your position of reasonability. Even if reasonable minds differ on the proposed approach, your position is not harmed – and probably strengthened – by attempting to confer with your opposition and achieve cooperation. If cooperation can be achieved, then you have furthered the case and saved your client money. If not, the judiciary is still the final arbiter and your attempt at cooperation will paint your proposed approach in the most reasonable light.

Using three document review examples, let’s see how technology and humans can be used and in what combination:

Door #1 – Linear review with 600 Reviewers

Many law firms and vendors have set up review shops with human reviewers or processes for linear review where every document is reviewed for a responsive call or privilege call. This method is also called the “eyes on every document” approach. It’s what has been done in document review for a very long time. Technology can be used with linear review to track your documents and decisions or to help tag or help cull documents to be reviewed because they meet a certain parameter (i.e. the selected set of custodians, date range or it comes back as a key term positive match). This method is time consuming and expensive.

Door #2 – Targeted Review with 60 reviewers

Corporations, law firms, and vendors also use a pared down approach employing technology to help with some aspects of the document review. This method can also be described as targeted review. Targeted review involves technology that will cluster or cull data into groups so that the data can be reviewed in a targeted, rather than linear, manner. Targeted review allows data to be coded in a threaded fashion for emails, or through groups of related documents in clusters, concepts, or batches. This method also involves some type of sampling to make sure that batch coding is being done correctly. This method doesn’t require a lot of set up on the front end and is time consuming during the actual review; but, efficiencies are built in to make it more efficient than linear review. Targeted review is also less costly than linear review because it can be done with fewer reviewers and reviewing “like” things together makes the review go faster.

Door #3 – Automated Review with 6 reviewers

Automated review involves more of a focus on technology with sophisticated input up front from the case team and subject matter experts. A spectrum of technology-based solutions can be deployed to automate as much or as little of the document review process as you want. Technology can involve the use of categorization or clustering and can involve linguistic or statistical methods to do so. This type of review is often referred to as automated review, technology assisted review, or predictive coding. (Although many use these terms interchangeably, they all mean different things to a linguistic or search and retrieval scientist.) This type of review can be done using a few reviewers and software, plus sampling to do the entire document review. Time and money efficiencies are built into this model. Note – The biggest change money-wise for this model is that the bulk of the money spent for document review is on the software or vendor running the software rather than the law firm or review team.

As you can see, no matter which door you choose, there is a mix of people and technology involved. Nowhere in any of these examples are humans left out of the picture! The question frequently asked is “Is technology replacing humans?” I think the answer is “No, just displacing low level human input.”

• It’s about efficiently and effectively deploying technology; Find the right tool for the right job whether it is culling, targeting, or segmenting data.
• It’s about efficiently and effectively deploying the people resources you have for your case; Find the right law firms, review firms, (outsourced or in-house), or vendors.

One quote I ran across recently made me pause: “In fact, many law firms are more comfortable hiring temporary reviewers or outsourcing the review to professional review attorney firms.” It’s a sad state of affairs if law firms are more comfortable outsourcing eDiscovery to foreign (US trained) lawyers or to contract/temporary attorneys rather than using a tool or technology that can eliminate the need for either one. Or worse yet, the law firms choose one tool for all eDiscovery services no matter what the size or content of the data is.

Looking at document review and how it involves humans and technology can illustrate the point that no matter what process you choose to review documents, it will be a mix of humans and technology. How it is mixed, is your choice. Human reviewers aren’t eliminated with automated reviews technology (or processes), although their presence is significantly reduced from the linear review model. The more sophisticated the technology used in document review, the more subject matter experts, case experts, and higher level expertise is required; which is a good thing because I don’t know anyone who went to law school to do document review as a living.

Lawyers will do almost anything to avoid having privileged documents turned over to the other side in a document production – this includes using (the very inefficient process of) linear review, which means trying to lay eyes on every single document in a document collection to make a privilege determination. Not only is that linear review process to determine (relevance and) privilege inefficient, but it will also cost a bundle which corporations are no longer willing to pay.

In the Oracle and Google patent case concerning Java, Google tried to keep out an email from Tim Lindholm and all 8 prior drafts. The document itself was marked privilege and put on a privilege log. The prior 8 drafts were not caught in Google’s searches for privileged documents, so they were not marked privilege or put on a privilege log.  

Google’s searches included searching the email address fields and searching for the words “Attorney Work Product.” The 8 drafts did not have an attorney listed in the To: field nor did they have the words Attorney Work Product on them (a privilege legend). (See page 9 of the Petition for Writ of Mandamus at http://en.calameo.com/read/00050025591a56e0c0fc5) Running that type of search for privilege meant that the earlier 8 drafts were not caught in the search for privileged documents. Those draft emails were turned over to the other side in the document productions.

Turning over these drafts could easily have been prevented using near duplicate technology to find any near duplicates for the documents to be withheld and listed on the privilege log. Just adding this last step alone would have found the 8 drafts of Tim Lindholm’s email message.

I’m hoping that the outside law firms Google involved here and Google itself was more sophisticated in its search approach than to just search for attorney names and for the words “attorney client privileged.” A well-thought out and properly constructed search strategy is paramount to finding the right information.

Searching for attorney names in the email address fields is ONE approach but it shouldn’t be your ONLY approach. Searching for the words “attorney work product” or “attorney client privileged” is ONE approach but it shouldn’t be your ONLY approach.  As Google found out the hard way, it shouldn’t be your only approach to just use these two methods to determine privilege. I’m not picking on Google (or their outside law firms) here, but this search was an incredibly simplistic approach to finding privileged documents – privilege search strategy should involve multiple parts: including searching for people (lawyers), law firm names, case names (or nicknames), the words like attorney or lawyer, and most importantly, the substance of the privilege being claimed.

There are many more sophisticated approaches to searching and reviewing these documents for privilege, but the real secret to finding privileged documents is to approach it from many angles.

Before diving into deep topics like statistical sampling or precision and recall, let’s start with something very basic. How do you know if a case will be a good fit for using Assisted Review? This is where my mantra Know your data comes into play.

1. How many of your documents are images or are mostly images such as PowerPoint slides? Assisted Review software relies on text, and can’t use images either for training the software or for assigning relevance tags. What about spreadsheets containing mostly numbers and formulas? Some software doesn’t recognize numbers as indexable items, plus the amount of variation in spreadsheet numbers makes categorization next to impossible for Assisted Review software. How many non-English or mixed-language documents are in your set? Some Assisted Review software can learn despite the language, though others cannot. Also, if your senior attorney can’t read the non-English languages, the attorney can’t indicate to the software if the document is relevant or not. You’ll need to tag these types of documents and prepare them for human review.
2. How many documents can you cull? One school of thought says you should never cull anything before using Assisted Review software, because the software needs the variation to learn what the not relevant documents look like. However, leaving in too many negative examples can lead to longer training cycles or require finding “seed” documents to teach it what relevant documents should look like. While to some extent this will depend on the case, de-NISTing, metadata culling for date and custodian, plus some simple keyword searching, will cut down on the non-responsive documents while still leaving enough variation for the software to be able to learn.
3. How much money will you save? At this point, you should have a set of documents belonging to your preferred custodians, within the date range that you’re concerned with, that are not solely or mostly images. Now is the time to take a hard look at the Assisted Review software you’re considering to use for this case. How much will it cost to run your remaining documents through the Assisted Review tool, compared to how much it will cost to run a first-level review? If you plan to review a subset of the documents returned as “most likely to be relevant” by the Assisted Review tool, add that cost in as well. Do the numbers work out?
4. How clearly is your definition of “relevant” defined? Even the best Assisted Review software isn’t a human being, and software can’t recognize some subtleties that humans can perceive. If you can define relevance for your case very clearly, in as binary a fashion as possible, then that definition will be easier for the software to learn than a long, drawn-out definition with many sub-parts. Be sure you’re as clear as possible, because this is the idea the software needs to learn.

These four questions should all be answered for every case you hope to use with Assisted Review software, before you start using the software. A little advance planning will help you identify the best cases to use with Assisted Review, which should save you time and money.

Technology assisted review is a process based around technology. Just as with “transparent search,” its results, in terms of cost and risk reduction, are as good as the people and process involved. With a clearly defined use-case, a strong iterative process, buy-in from both counsel/parties, a team of experts to train the tool, and verification of the results, it is possible to run a review cost-effectively and with better results, i.e., with both higher precision and higher recall. That’s the good news. The bad news is that much confusion still exists. I attended a panel titled ‘eDiscovery Process Validation Sampling – How Confident Can You Be in the Results of Your Process?’ It was clear early on that none of the panelists understood the topic – in fact, they pretty much avoided it altogether. In addition, the industry is using technology assisted review in many different matters, and at many different stages of the process, and for various different reasons. When these mists clear the adoption rate will grow dramatically.