Merge branch '2_1_5_reverse_merge'

Author: beebs-systap

LEGAL/aduna-bsd.txt

@@ -0,0 +1,25 @@
+Copyright Aduna (http://www.aduna-software.com/) 2001-2013
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification,
+are permitted provided that the following conditions are met:
+
+    * Redistributions of source code must retain the above copyright notice,
+      this list of conditions and the following disclaimer.
+    * Redistributions in binary form must reproduce the above copyright notice,
+      this list of conditions and the following disclaimer in the documentation
+      and/or other materials provided with the distribution.
+    * Neither the name of the copyright holder nor the names of its contributors
+      may be used to endorse or promote products derived from this software
+      without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
+ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
+ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

README.md

@@ -1,5 +1,5 @@
-## Welcome to the Blazegraph Database
-Blazegraph™ DB is a ultra high-performance graph database supporting Blueprints and RDF/SPARQL APIs. It supports up to 50 Billion edges on a single machine and has a enterprise features for High Availability, Scale-out architecture, and [GPU Acceleration](https://www.blazegraph.com/product/gpu-accelerated/). It is in production use for Fortune 500 customers such as EMC, Autodesk, and many others.  It is supporting key [Precision Medicine](http://www.syapse.com) applications and has wide-spread usage for life science applications.  It is used extensively to support Cyber anaytics in commercial and government applications.  It powers the Wikimedia Foundation's [Wiki Data Query Service](https://query.wikidata.org/).  See the latest [Feature Matrix](http://www.blazegraph.com/product/).
+## Welcome to the Blazegraph Database ##
+Blazegraph™ DB is a ultra high-performance graph database supporting Blueprints and RDF/SPARQL APIs. It supports up to 50 Billion edges on a single machine. It is in production use for Fortune 500 customers such as EMC, Autodesk, and many others.  It is supporting key [Precision Medicine](http://www.syapse.com) applications and has wide-spread usage for life science applications.  It is used extensively to support Cyber anaytics in commercial and government applications.  It powers the Wikimedia Foundation's [Wikidata Query Service](https://query.wikidata.org/). 
 
 ![image](http://blog.blazegraph.com/wp-content/uploads/2015/07/blazegraph_by_systap_favicon.png)
 
@@ -9,20 +9,47 @@ Please see the release notes in [releases](bigdata/src/releases) for version cha
 
 Please also visit us at our: [website](https://www.blazegraph.com), [wiki](https://wiki.blazegraph.com), and [blog](https://blog.blazegraph.com/).
 
-Find an issue?   Need help?  See [JIRA](https://jira.blazegraph.com) or purchase [Support](https://www.blazegraph.com/buy).
+Find an issue?   Need help?  See [JIRA](https://jira.blazegraph.com).
 
 Reporting a security issue: [Security Reporting](Security.md).
 
-###Quick Start with the Executable Jar
+### Quick Start with the Executable Jar ###
 Up and running with Blazegraph in under 30 seconds:  [Quick Start](https://wiki.blazegraph.com/wiki/index.php/Quick_Start).
 
-###Samples and Examples
+### Deploying in Production ###
+Blazegraph is designed to be easy to use and get started. It ships without SSL or authentication by default for this reason. For production deployments, we _strongly_ recommend you enable SSL, authentication, and appropriate network configurations. There are some helpful links below to enable you to do this. 
+
+#### Enabling SSL support ####
+To enable SSL support, uncomment the example [jetty.xml](blazegraph-jar/src/main/resources/jetty.xml#L141) and configure it for your local keystore.
+
+#### Configuration Authentication ####
+By default, Blazegraph ships without authentication enabled. This is great for developing, getting started, and doing research with Blazegraph. However, it's not recommended for any production deployment. To configuration authentication, you must configure it either within the web app container or via a reverse-proxy configuration.
+
+Note that the Blazegraph namespace feature for [multi-tenancy](https://wiki.blazegraph.com/wiki/index.php/REST_API#Multi-Tenancy_API) does not provide security isolation. Users that can access the base URI of the server can access any of the available namespaces. You can further restrict this through a combination of authentication configuration and restricting access to specific namespace URIs, i.e. `/blazegraph/namespace/NAMESPACE/sparql`.
+
+There are three basic options:
+
+1. **Configuring Jetty Authentication for a standalone Jetty deployment**:  Follow the [jetty](http://www.eclipse.org/jetty/documentation/9.2.22.v20170531/configuring-security-authentication.html) guide to configure authentication for the [jetty.xml](blazegraph-jar/src/main/resources/jetty.xml) you use to deploy the server by uncommenting the `<Get name="securityHandler">` section. You'll need to create a [realm.properties](blazegraph-jar/src/main/resources/realm.properties) and update the jetty.xml to point to its location on the filesystem.  Then configure the [web.xml](bigdata-war-html/src/main/webapp/WEB-INF/web.xml) to uncomment the security-constraint.
+1. **Configuring Tomcat Authentication for a standalone Tomcat deployment**:  First configure a Tomcat [Realm](https://tomcat.apache.org/tomcat-7.0-doc/realm-howto.html) with your choice of authentication method (JDBC, JNDI, etc.). Then configure the [web.xml](bigdata-war-html/src/main/webapp/WEB-INF/web.xml) to uncomment the security-constraint.
+1. **Setup a reverse-proxy configuration with authentication**:  You can setup an http or https reverse proxy configuration that has authentication and forward requests to the local Blazegraph instance (typically running on localhost:9999). This is a good option with [Nginx](https://community.openhab.org/t/using-nginx-reverse-proxy-authentication-and-https/14542) and [Apache](https://stackoverflow.com/questions/5011102/apache-reverse-proxy-with-basic-authentication). 
+
+##### Mitigating Cross-Site Request Forgery (CSRF) #####
+If you enable authentication and expose the Blazegraph workbench, you should also take steps to protect against CSRF. Tomcat8 provides a [CSRF filter](https://tomcat.apache.org/tomcat-8.0-doc/config/filter.html#CSRF_Prevention_Filter_for_REST_APIs) that can be configured. For Jetty, if you configure authentication the default value for `SecurityHandler.setSessionRenewedOnAuthentication(true)` can also be used. CSRF protection may require REST clients to implement HTTP headers to be used to interact with the service.
+
+### Building the code ###
+As a quick start, run `mvn install -DskipTests` or the utility script `./scripts/mavenInstall.sh `.
+
+For more detailed maven information see the [wiki](https://wiki.blazegraph.com/wiki/index.php/MavenNotes). 
+
+If you build with Java 7, you need to add Maven options for TLS 1.2, i.e. `export MAVEN_OPTS="-Dhttps.protocols=TLSv1.2"`.
+
+### Samples and Examples ###
 There are code samples and examples to get started with the Blazegraph Database [here] (https://github.com/blazegraph/blazegraph-samples).  Tinkerpop3 examples are included directly within the Tinkerpop3 repository per below.
 
-###Javadocs
+### Javadocs ###
 Click here to view the lastest [API Javadocs](https://blazegraph.github.io/database/apidocs/index.html).
 
-###Maven Central
+### Maven Central ###
 Starting with the 2.0.0 release, the Blazegraph Database is available on Maven Central.  To include the core platform and dependencies, include the artifact below in your dependencies.   [Developing with Maven](https://wiki.blazegraph.com/wiki/index.php/MavenNotes) has notes on developing with Blazegraph Database source code and Maven.
 
 ```
@@ -49,13 +76,13 @@ If you'd just link the Blazegraph Database dependencies without any of the exter
     </dependency>
 ```
 
-###Deployers
+### Deployers ###
 
 Starting with 2.0.0, the default context path for deployment is `http://localhost:9999/blazegraph/`.  There are also Maven artifacts for WAR deployers (`blazegraph-war`), executable Jar files (`blazegraph-jar`), [Debian Package](blazegraph-deb/) (`blazegraph-deb`), [RPM](blazegraph-rpm/) (`blazegraph-rpm`), and a [Tarball](blazegraph-tgz/) (`blazegraph-tgz`).
 
 The `bigdata-war` and `bigdata-jar` artifacts are included for legacy purposes and use the `/bigdata/` context path.
 
-###Tinkerpop3
+### Tinkerpop3 ###
 Tinkerpop3 supports requires Java 1.8 and is now in a separate repository.  See [Tinkerpop3](https://github.com/blazegraph/tinkerpop3).  It is also available as Maven Central artifact.
 
 ```
@@ -67,19 +94,11 @@ Tinkerpop3 supports requires Java 1.8 and is now in a separate repository.  See
     
 ```
 
-###Triple Pattern Fragment (TPF) Server
-There is a [Triple Pattern Fragment (TPF) for Blazegraph](https://github.com/TPF4Blazegraph/TPF4Blazegraph) server that supports [Linked Data Fragments](http://linkeddatafragments.org/).
-
-```
-    <dependency>
-        <groupId>com.blazegraph</groupId>
-        <artifactId>BlazegraphBasedTPFServer</artifactId>
-        <version>0.1.0</version>
-    </dependency>
-```    
+### Triple Pattern Fragment (TPF) Server ###
+There is a [Blazegraph Triple Pattern Fragment TPF](https://github.com/hartig/BlazegraphBasedTPFServer) server that supports [Linked Data Fragments](http://linkeddatafragments.org/).
 
-###Blazegraph Python Client
-There is a Blazegraph Python Client [here] (https://github.com/blazegraph/blazegraph-python)
+### Blazegraph Python Client ###
+There is a Blazegraph Python Client [here](https://github.com/blazegraph/blazegraph-python)
 
-###Blazegraph Dot Net RDF Client
+### Blazegraph Dot Net RDF Client ###
 There is a Blazegraph Dot Net RDF Client [here](https://github.com/blazegraph/blazegraph-dotnetrdf)

Security.md

@@ -1,10 +1,10 @@
-##Reporting a Blazegraph Security Issue##
+## Reporting a Blazegraph Security Issue ##
 
 For customers and Blazegraph users, please send an email to security [at] blazegraph.com to report a security issue. You may send an encrypted message using the public key below.
 
 This alias is monitored on a daily basis.   All security reports are acknowledged within 24 hours.   Mitigations for reported security issues are made in a reasonable timeframe, which may be as quickly as 24 hours for high-severity issues.
 
-###Public Key###
+### Public Key ###
 Please use the public key below for transmitting any sensitive information to the security alias.
 
 ```

bigdata-blueprints/pom.xml

@@ -28,12 +28,12 @@ Copyright 2010 by TalkingTrends (Amsterdam, The Netherlands)
   <parent>
     <groupId>com.blazegraph</groupId>
     <artifactId>blazegraph-parent</artifactId>
-    <version>2.1.4</version>
+    <version>2.1.6-SNAPSHOT</version>
     <relativePath>../pom.xml</relativePath>
   </parent>
   <groupId>com.blazegraph</groupId>
   <artifactId>bigdata-blueprints</artifactId>
-  <version>2.1.4</version>
+  <version>2.1.6-SNAPSHOT</version>
   <name>Blazegraph Blueprints API</name>
   <description>Blazegraph support for Tinkerpop 2.5</description>
   <packaging>jar</packaging>

bigdata-cache/pom.xml

@@ -28,12 +28,12 @@ Copyright 2010 by TalkingTrends (Amsterdam, The Netherlands)
   <parent>
     <groupId>com.blazegraph</groupId>
     <artifactId>blazegraph-parent</artifactId>
-    <version>2.1.4</version>
+    <version>2.1.6-SNAPSHOT</version>
     <relativePath>../pom.xml</relativePath>
   </parent>
   <groupId>com.blazegraph</groupId>
   <artifactId>bigdata-cache</artifactId>
-  <version>2.1.4</version>
+  <version>2.1.6-SNAPSHOT</version>
   <name>Blazegraph Cache</name>
   <description>Blazegraph Cache utilities</description>
   <packaging>jar</packaging>

bigdata-client/pom.xml

@@ -28,12 +28,12 @@ Copyright 2010 by TalkingTrends (Amsterdam, The Netherlands)
   <parent>
     <groupId>com.blazegraph</groupId>
     <artifactId>blazegraph-parent</artifactId>
-    <version>2.1.4</version>
+    <version>2.1.6-SNAPSHOT</version>
     <relativePath>../pom.xml</relativePath>
   </parent>
   <groupId>com.blazegraph</groupId>
   <artifactId>bigdata-client</artifactId>
-  <version>2.1.4</version>
+  <version>2.1.6-SNAPSHOT</version>
   <name>Blazegraph Client API</name>
   <description>Blazegraph Client API tools</description>
   <build>

bigdata-client/src/main/java/com/bigdata/rdf/sail/webapp/client/AST2SPARQLUtil.java

@@ -34,6 +34,8 @@
 import org.openrdf.model.Literal;
 import org.openrdf.model.URI;
 import org.openrdf.model.Value;
+import org.openrdf.model.vocabulary.XMLSchema;
+import org.openrdf.query.parser.sparql.SPARQLUtil;
 
 /**
  * Utility class for externalizing SPARQL prefix declaration management.
@@ -154,7 +156,7 @@ public String toExternal(final Literal lit) {
                 + (datatypeURI != null ? datatypeStr.length() + 2 : 0));
 
         sb.append('"');
-        sb.append(label);
+        sb.append(SPARQLUtil.encodeString(label));
         sb.append('"');
 
         if (languageCode != null) {

bigdata-client/src/test/java/com/bigdata/rdf/sail/webapp/client/MockRemoteRepository.java

@@ -12,6 +12,7 @@
 import org.eclipse.jetty.client.api.Response.ResponseListener;
 import org.eclipse.jetty.client.api.Result;
 import org.eclipse.jetty.http.HttpHeader;
+import org.eclipse.jetty.util.Callback;
 import org.openrdf.query.resultio.TupleQueryResultFormat;
 import org.openrdf.rio.RDFFormat;
 
@@ -34,7 +35,7 @@ public static MockRemoteRepository create(final String tupleQueryResponse, final
 		// pojo to retrieve values from mock service
 		final Data data = new Data();
 
-		String serviceURL = "localhost";
+		String serviceURL = "http://localhost";
 		HttpClient httpClient = new HttpClient() {
 			@Override
 			protected void send(HttpRequest request,
@@ -67,7 +68,7 @@ public int getStatus() {
 						java.nio.ByteBuffer buf = java.nio.ByteBuffer.allocate(responseContent.length());
 						buf.put(responseContent.getBytes(Charset.forName(StandardCharsets.UTF_8.name())));
 						buf.flip();
-						((JettyResponseListener)listener).onContent(response, buf);
+						((JettyResponseListener)listener).onContent(response, buf, Callback.NOOP);
 						((JettyResponseListener)listener).onSuccess(response);
 						((JettyResponseListener)listener).onComplete(new Result(request, response));
 					}

bigdata-common-util/pom.xml

@@ -28,12 +28,12 @@ Copyright 2010 by TalkingTrends (Amsterdam, The Netherlands)
   <parent>
     <groupId>com.blazegraph</groupId>
     <artifactId>blazegraph-parent</artifactId>
-    <version>2.1.4</version>
+    <version>2.1.6-SNAPSHOT</version>
     <relativePath>../pom.xml</relativePath>
   </parent>
   <groupId>com.blazegraph</groupId>
   <artifactId>bigdata-common-util</artifactId>
-  <version>2.1.4</version>
+  <version>2.1.6-SNAPSHOT</version>
   <name>Blazegraph Common Utilities</name>
   <description>Blazegraph utilities common across artifacts</description>
   <build>
@@ -113,7 +113,7 @@ ga('send', 'pageview');
     </dependency>
     <dependency>
       <groupId>com.blazegraph</groupId>
-      <artifactId>colt</artifactId>
+      <artifactId>blazegraph-colt</artifactId>
       <version>${colt.version}</version>
     </dependency>
     <dependency>

bigdata-core-test/bigdata/src/test/com/bigdata/htree/TestHTree.java

@@ -27,14 +27,24 @@
 
 package com.bigdata.htree;
 
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Random;
+import java.util.Set;
+
+import org.apache.commons.lang3.tuple.Pair;
+
 import com.bigdata.btree.AbstractBTreeTestCase;
 import com.bigdata.btree.ITuple;
 import com.bigdata.btree.ITupleIterator;
 import com.bigdata.btree.raba.ReadOnlyKeysRaba;
 import com.bigdata.btree.raba.ReadOnlyValuesRaba;
 import com.bigdata.htree.data.MockBucketData;
+import com.bigdata.io.ByteArrayBuffer;
 import com.bigdata.rawstore.IRawStore;
 import com.bigdata.rawstore.SimpleMemoryRawStore;
+import com.bigdata.util.BytesUtil;
 
 /**
  * Unit tests for {@link HTree}.
@@ -857,5 +867,97 @@ public void test_distinctBits() {
         }
 
     }
-    
+	
+	
+	/**
+	 * Insert data into the BTRee, making sure that data previously
+	 * inserted is found in later stages when looking it up.
+	 */
+	public void testInsertAndLookupDups() {
+
+        final int addressBits = 2;
+        final IRawStore store = new SimpleMemoryRawStore();
+		final HTree htree = getHTree(store, addressBits, true, true);
+		
+		final Random rand = new Random(0);
+		final List<Pair<byte[],byte[]>> keyValuePairs = new LinkedList<>();
+		
+		final Set<Integer> insertedKeys = new HashSet<Integer>();
+		for (int i=0; i<100000; i++) {
+			
+			final int idx = nextKeyValuePair(keyValuePairs, rand);
+			final Pair<byte[],byte[]> keyValuePair = keyValuePairs.get(idx);
+
+			// did we see this one before?
+			final boolean isDuplicate = insertedKeys.contains(idx);
+
+			final byte[] key = keyValuePair.getLeft();
+			final byte[] val = keyValuePair.getRight();
+
+
+			if (isDuplicate) {
+
+				// validate that the key value pair is found in the HTree
+				boolean confirmed = false;
+				
+				
+				final ITupleIterator<?> titr = htree.lookupAll(key);
+				
+	            while (titr.hasNext()) {
+
+					final ITuple<?> t = titr.next();
+
+					final ByteArrayBuffer tb = t.getValueBuffer();
+
+					if (0 == BytesUtil.compareBytesWithLenAndOffset(
+							0/* aoff */, val.length/* alen */, val, //
+							0/* boff */, tb.limit()/* blen */, tb.array()/* b */
+					)) {
+
+						confirmed = true;
+						break;
+
+					}
+				}
+
+	            if (!confirmed) {
+	            	throw new RuntimeException("Expected to find, but did not succeed");
+	            }
+
+			} else {
+				
+				htree.insert(key, val); // insert into HTree
+				insertedKeys.add(idx);	// remember as duplicate for next encounter
+				
+			}			
+		}
+	}	
+	
+	
+	/**
+	 * Randomly either expends the keyValuePairs list by an element and
+	 * returns the index of the last list position OR returns an index
+	 * of a known position (i.e., a duplicate). Duplicates will be returned
+	 * about every 10th call.
+	 */
+	private int nextKeyValuePair(final List<Pair<byte[],byte[]>> keyValuePairs, final Random rand) {
+		
+		// usually we generate a new key-value pair and append it to the index
+		if (keyValuePairs.isEmpty() || rand.nextInt(10) != 0) {
+			final byte[] key = new byte[4];
+			final byte[] value = new byte[18];
+			
+			rand.nextBytes(key);
+			rand.nextBytes(value);
+			
+			keyValuePairs.add(Pair.of(key, value));
+			
+			return keyValuePairs.size()-1; // idx of generated value
+			
+		} else { // !keyValuePairs.isEmpty() && rand.nextInt()==10
+			
+			// return a tripe from before
+			return rand.nextInt(keyValuePairs.size());
+		}
+	}
 }