Compare commits

..

66 Commits

Author SHA1 Message Date
Ceda EI 20fb7f9bc2 Merge upstream new version 2021-08-25 00:16:57 +05:30
John Crepezzi 5d2965ffc5
Merge pull request #350 from seejohnrun/specify-config-on-boot
Allow setting config.js alternative on boot
2020-10-06 22:15:59 -04:00
John Crepezzi f255928af7 Allow setting config.js alternative on boot
Closes #105
2020-10-06 22:15:13 -04:00
John Crepezzi a108dbadc5
Merge pull request #349 from seejohnrun/add-head-support
Add support for HEAD requests
2020-10-06 22:07:19 -04:00
John Crepezzi c409aca080 Add support for HEAD requests
On regular document endpoints, and on raw endpoints
2020-10-06 22:05:22 -04:00
John Crepezzi 219424550b Fix local name 2020-10-06 21:02:16 -04:00
John Crepezzi f147acb51c Switch to using pg.Pool 2020-10-06 21:01:14 -04:00
John Crepezzi 9a692ed652 Get the client working as expected with pg 8 2020-10-06 21:01:10 -04:00
John Crepezzi 3a17c86a0f Upgrade pg to the most recent version
This isn't going to actually work yet, just getting things in place
2020-10-06 21:01:01 -04:00
John Crepezzi 677a22987a
Merge pull request #122 from Roundaround/mongodb
Added mongodb document store adapter
2020-10-06 01:46:59 -04:00
John Crepezzi 89d912c6ff
Merge pull request #347 from seejohnrun/fix-memcached
Fix memcached client fetch for key not found
2020-10-06 01:39:52 -04:00
John Crepezzi 4cac6713ef Fix memcached client fetch for key not found
The memcached client wasn't correctly handling looking up a key that
didn't exist.  Now we only try to push the expiration forward if there
is actually a value in memcached.

Also while I'm in here, allow expiration to be left blank.
2020-10-06 01:36:46 -04:00
John Crepezzi f3b0de745b
Merge pull request #200 from kevinhaendel/master
Added "user-select" option to line numbers & messages
2020-10-06 01:21:16 -04:00
John Crepezzi cc8a99752f
Merge pull request #271 from mklkj/fix-content-type
Fix contentType header in save request
2020-10-06 01:18:34 -04:00
John Crepezzi 6853d077e7 Merge branch 'master' into fix-content-type 2020-10-06 01:18:22 -04:00
John Crepezzi 80a2b6f0dd
Merge pull request #241 from meseta/master
Add Google Datastore sorage handler
2020-10-06 01:15:24 -04:00
John Crepezzi 4f68b3d7d6 Merge branch 'master' into meseta/master 2020-10-06 01:13:07 -04:00
John Crepezzi ef0ca40533 Downgrade pg for now
Will make a PR to use the new APIs soon
2020-10-06 00:54:12 -04:00
John Crepezzi f372ef18de
Merge pull request #345 from seejohnrun/fix-json-highlighting
Use the now-separate json mode for json highlighting
2020-10-06 00:37:21 -04:00
John Crepezzi 181a3a2bfa Use the now-separate json mode for json highlighting
Closes #267
2020-10-06 00:36:41 -04:00
John Crepezzi 61d08afb3b
Merge pull request #344 from seejohnrun/upgrade-highlight-js
Update highlight JS to the most recent version (10.2.1)
2020-10-06 00:10:59 -04:00
John Crepezzi 1ba025328d Update highlight JS to the most recent version 2020-10-06 00:07:07 -04:00
John Crepezzi a79fb39f54 Merge branch 'master' of github.com:seejohnrun/haste-server 2020-10-05 23:52:13 -04:00
John Crepezzi 3a72d74537 Fix security vulnerabilities from outdated packages
Closes #258
2020-10-05 23:50:28 -04:00
John Crepezzi e9ae74b7a9
Merge pull request #322 from sethsmoe/patch-1
remove 1px margin from textarea, fixes useless scrollbar
2020-09-22 15:42:35 -04:00
John Crepezzi c305e9a83d
Merge pull request #342 from seejohnrun/dependabot/npm_and_yarn/bl-4.0.3
Bump bl from 4.0.2 to 4.0.3
2020-09-22 15:41:12 -04:00
dependabot[bot] 16bce4c83d
Bump bl from 4.0.2 to 4.0.3
Bumps [bl](https://github.com/rvagg/bl) from 4.0.2 to 4.0.3.
- [Release notes](https://github.com/rvagg/bl/releases)
- [Commits](https://github.com/rvagg/bl/compare/v4.0.2...v4.0.3)

Signed-off-by: dependabot[bot] <support@github.com>
2020-09-22 19:40:54 +00:00
John Crepezzi 661997cd73
Merge pull request #334 from ourforks/master
[Security] Update dependencies to reduce risk
2020-09-22 15:40:27 -04:00
John Crepezzi 159f989d08
Merge pull request #335 from emillen/docker-support
Docker support
2020-09-22 15:39:20 -04:00
emil-lengman 139df62ec4 add newline to stop github complaining 2020-08-22 22:27:05 +02:00
emil-lengman bae6387bb7 forgot to rename some vars 2020-08-22 22:25:10 +02:00
emil-lengman bb7b9571a7 write some documentation for the Docker solution 2020-08-22 22:22:08 +02:00
emil-lengman a4dc29fb2b its supposed to be milliseconds 2020-08-22 22:12:54 +02:00
emil-lengman 342f56ce1a use same password and username env vars for all types 2020-08-22 21:56:58 +02:00
emil-lengman 05ecc90764 add file path 2020-08-22 21:47:48 +02:00
emil-lengman 69cf505a90 remove pg connect string, add rethink user and password 2020-08-22 21:29:41 +02:00
emil-lengman 9f41993566 also install rethinkdb and aws-sdk 2020-08-22 21:06:02 +02:00
emil-lengman 5c9311fb85 remove unused import 2020-08-22 20:56:45 +02:00
emil-lengman 5a8d52a5e3 add healthcheck, and stopsignal, plus export the correct port 2020-08-22 20:56:11 +02:00
emil-lengman 0f145b4444 pin versions 2020-08-22 20:48:32 +02:00
emil-lengman aef4bb5edb add dockerignore file 2020-08-22 20:45:27 +02:00
emil-lengman 36c854ef1b move creating the config file to a js file 2020-08-22 20:44:32 +02:00
emil-lengman edd428ff37 fix some names for env vars 2020-08-22 20:43:39 +02:00
emil-lengman 0612ba001e basic docker-compose for running the project together with memcached 2020-08-22 17:33:40 +02:00
emil-lengman 064680003d basic dockerfile with default env vars 2020-08-22 17:33:19 +02:00
emil-lengman 655f2af45a script for turning env-vars into config.js 2020-08-22 17:33:01 +02:00
Reece Dunham ce03749c2f Update dependencies to reduce security risk
Signed-off-by: Reece Dunham <me@rdil.rocks>
2020-08-12 23:45:18 +00:00
epdn f6084b4339
remove 1px margin from textarea, fixes useless scrollbar 2020-05-18 09:34:57 +01:00
John Crepezzi 9b0a5ff0a3
Merge pull request #291 from j3parker/s3-document-store
Add an Amazon S3 document store
2020-02-27 11:39:12 -05:00
Jacob Parker 1fff48568f Document the IAM permissions 2019-07-08 16:59:04 +01:00
Jacob Parker b4c666fbcf Add an Amazon S3 document store 2019-06-28 19:25:49 +01:00
Mikołaj Pich 035cf0e91e
Fix content type 2018-12-22 15:11:37 +01:00
Yuan Gao 86bbc1899d
Update README.md 2018-09-01 21:12:30 +01:00
Yuan Gao d41d7491d4
rename to google-datastore, and use Date.now() 2018-09-01 21:11:58 +01:00
Yuan 5fb43eb67c added condition for this.expire not defined 2018-08-28 01:28:26 +01:00
Yuan 1eeef4ede4 restored using null 2018-08-28 01:21:37 +01:00
Yuan ebc749c5e0 updated readme 2018-08-28 00:37:21 +01:00
Yuan b0bbb72f35 updated to use Date(null) 2018-08-28 00:35:09 +01:00
Yuan 2213c3874a updated readme 2018-08-27 23:48:19 +01:00
Yuan 6ebd72a86c updated readme 2018-08-27 23:34:56 +01:00
Yuan b6814a1445 bugfixes 2018-08-27 23:15:02 +01:00
Yuan e3d18efdc6 added npm package 2018-08-27 23:01:37 +01:00
Yuan 869fb65738 added googledatastore handler 2018-08-27 22:59:05 +01:00
Yuan Gao 56b939124e
Merge pull request #2 from seejohnrun/master
update from source
2018-08-27 22:58:29 +01:00
Kevin Händel 16d529e935
Added "user-select" option to line numbers & messages
This prevents copying unnecessary text after selecting it via Ctrl + A
2018-02-11 00:35:27 +01:00
Evan Steinkerchner d3db5e2a5d Added mongodb document store adapter 2016-06-10 16:43:43 -04:00
20 changed files with 1999 additions and 273 deletions

6
.dockerignore Normal file
View File

@ -0,0 +1,6 @@
npm-debug.log
node_modules
*.swp
*.swo
data
*.DS_Store

63
Dockerfile Normal file
View File

@ -0,0 +1,63 @@
FROM node:14.8.0-stretch
RUN mkdir -p /usr/src/app && \
chown node:node /usr/src/app
USER node:node
WORKDIR /usr/src/app
COPY --chown=node:node . .
RUN npm install && \
npm install redis@0.8.1 && \
npm install pg@4.1.1 && \
npm install memcached@2.2.2 && \
npm install aws-sdk@2.738.0 && \
npm install rethinkdbdash@2.3.31
ENV STORAGE_TYPE=memcached \
STORAGE_HOST=127.0.0.1 \
STORAGE_PORT=11211\
STORAGE_EXPIRE_SECONDS=2592000\
STORAGE_DB=2 \
STORAGE_AWS_BUCKET= \
STORAGE_AWS_REGION= \
STORAGE_USENAMER= \
STORAGE_PASSWORD= \
STORAGE_FILEPATH=
ENV LOGGING_LEVEL=verbose \
LOGGING_TYPE=Console \
LOGGING_COLORIZE=true
ENV HOST=0.0.0.0\
PORT=7777\
KEY_LENGTH=10\
MAX_LENGTH=400000\
STATIC_MAX_AGE=86400\
RECOMPRESS_STATIC_ASSETS=true
ENV KEYGENERATOR_TYPE=phonetic \
KEYGENERATOR_KEYSPACE=
ENV RATELIMITS_NORMAL_TOTAL_REQUESTS=500\
RATELIMITS_NORMAL_EVERY_MILLISECONDS=60000 \
RATELIMITS_WHITELIST_TOTAL_REQUESTS= \
RATELIMITS_WHITELIST_EVERY_MILLISECONDS= \
# comma separated list for the whitelisted \
RATELIMITS_WHITELIST=example1.whitelist,example2.whitelist \
\
RATELIMITS_BLACKLIST_TOTAL_REQUESTS= \
RATELIMITS_BLACKLIST_EVERY_MILLISECONDS= \
# comma separated list for the blacklisted \
RATELIMITS_BLACKLIST=example1.blacklist,example2.blacklist
ENV DOCUMENTS=about=./about.md
EXPOSE ${PORT}
STOPSIGNAL SIGINT
ENTRYPOINT [ "bash", "docker-entrypoint.sh" ]
HEALTHCHECK --interval=30s --timeout=30s --start-period=5s \
--retries=3 CMD [ "curl" , "-f" "localhost:${PORT}", "||", "exit", "1"]
CMD ["npm", "start"]

160
README.md
View File

@ -31,7 +31,7 @@ STDOUT. Check the README there for more details and usages.
1. Download the package, and expand it
2. Explore the settings inside of config.js, but the defaults should be good
3. `npm install`
4. `npm start`
4. `npm start` (you may specify an optional `<config-path>` as well)
## Settings
@ -52,7 +52,7 @@ STDOUT. Check the README there for more details and usages.
When present, the `rateLimits` option enables built-in rate limiting courtesy
of `connect-ratelimit`. Any of the options supported by that library can be
used and set in `config.json`.
used and set in `config.js`.
See the README for [connect-ratelimit](https://github.com/dharmafly/connect-ratelimit)
for more information!
@ -154,6 +154,28 @@ or post.
All of which are optional except `type` with very logical default values.
### MongoDB
To use mongodb storage you must install the 'mongodb' pachage in npm
`npm install mongodb`
Once you've done that, your config section should look like:
``` json
{
"type": "mongodb",
"connectionUrl": "mongodb://localhost:27017/database"
}
```
You can also just set the environment variable for `DATABASE_URL` to your database connection url.
Unlike with postgres you do NOT have to create the table in your mongo database prior to running.
You can also set an `expire` option to the number of seconds to expire keys in.
This is off by default, but will constantly kick back expirations on each view or post.
### Memcached
To use memcache storage you must install the `memcached` package via npm
@ -198,6 +220,140 @@ Also, you must create an `uploads` table, which will store all the data for uplo
You can optionally add the `user` and `password` properties to use a user system.
### Google Datastore
To use the Google Datastore storage system, you must install the `@google-cloud/datastore` package via npm
`npm install @google-cloud/datastore`
Once you've done that, your config section should look like this:
``` json
{
"type": "google-datastore"
}
```
Authentication is handled automatically by [Google Cloud service account credentials](https://cloud.google.com/docs/authentication/getting-started), by providing authentication details to the GOOGLE_APPLICATION_CREDENTIALS environmental variable.
### Amazon S3
To use [Amazon S3](https://aws.amazon.com/s3/) as a storage system, you must
install the `aws-sdk` package via npm:
`npm install aws-sdk`
Once you've done that, your config section should look like this:
```json
{
"type": "amazon-s3",
"bucket": "your-bucket-name",
"region": "us-east-1"
}
```
Authentication is handled automatically by the client. Check
[Amazon's documentation](https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/setting-credentials-node.html)
for more information. You will need to grant your role these permissions to
your bucket:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::your-bucket-name-goes-here/*"
}
]
}
```
## Docker
### Build image
```bash
docker build --tag haste-server .
```
### Run container
For this example we will run haste-server, and connect it to a redis server
```bash
docker run --name haste-server-container --env STORAGE_TYPE=redis --env STORAGE_HOST=redis-server --env STORAGE_PORT=6379 haste-server
```
### Use docker-compose example
There is an example `docker-compose.yml` which runs haste-server together with memcached
```bash
docker-compose up
```
### Configuration
The docker image is configured using environmental variables as you can see in the example above.
Here is a list of all the environment variables
### Storage
| Name | Default value | Description |
| :--------------------: | :-----------: | :-----------------------------------------------------------------------------------------------------------: |
| STORAGE_TYPE | memcached | Type of storage . Accepted values: "memcached","redis","postgres","rethinkdb", "amazon-s3", and "file" |
| STORAGE_HOST | 127.0.0.1 | Storage host. Applicable for types: memcached, redis, postgres, and rethinkdb |
| STORAGE_PORT | 11211 | Port on the storage host. Applicable for types: memcached, redis, postgres, and rethinkdb |
| STORAGE_EXPIRE_SECONDS | 2592000 | Number of seconds to expire keys in. Applicable for types. Redis, postgres, memcached. `expire` option to the |
| STORAGE_DB | 2 | The name of the database. Applicable for redis, postgres, and rethinkdb |
| STORAGE_PASSWORD | | Password for database. Applicable for redis, postges, rethinkdb . |
| STORAGE_USERNAME | | Database username. Applicable for postgres, and rethinkdb |
| STORAGE_AWS_BUCKET | | Applicable for amazon-s3. This is the name of the S3 bucket |
| STORAGE_AWS_REGION | | Applicable for amazon-s3. The region in which the bucket is located |
| STORAGE_FILEPATH | | Path to file to save data to. Applicable for type file |
### Logging
| Name | Default value | Description |
| :---------------: | :-----------: | :---------: |
| LOGGING_LEVEL | verbose | |
| LOGGING_TYPE= | Console |
| LOGGING_COLORIZE= | true |
### Basics
| Name | Default value | Description |
| :----------------------: | :--------------: | :---------------------------------------------------------------------------------------: |
| HOST | 0.0.0.0 | The hostname which the server answers on |
| PORT | 7777 | The port on which the server is running |
| KEY_LENGTH | 10 | the length of the keys to user |
| MAX_LENGTH | 400000 | maximum length of a paste |
| STATIC_MAX_AGE | 86400 | max age for static assets |
| RECOMPRESS_STATIC_ASSETS | true | whether or not to compile static js assets |
| KEYGENERATOR_TYPE | phonetic | Type of key generator. Acceptable values: "phonetic", or "random" |
| KEYGENERATOR_KEYSPACE | | keySpace argument is a string of acceptable characters |
| DOCUMENTS | about=./about.md | Comma separated list of static documents to serve. ex: \n about=./about.md,home=./home.md |
### Rate limits
| Name | Default value | Description |
| :----------------------------------: | :-----------------------------------: | :--------------------------------------------------------------------------------------: |
| RATELIMITS_NORMAL_TOTAL_REQUESTS | 500 | By default anyone uncategorized will be subject to 500 requests in the defined timespan. |
| RATELIMITS_NORMAL_EVERY_MILLISECONDS | 60000 | The timespan to allow the total requests for uncategorized users |
| RATELIMITS_WHITELIST_TOTAL_REQUESTS | | By default client names in the whitelist will not have their requests limited. |
| RATELIMITS_WHITELIST_EVERY_SECONDS | | By default client names in the whitelist will not have their requests limited. |
| RATELIMITS_WHITELIST | example1.whitelist,example2.whitelist | Comma separated list of the clients which are in the whitelist pool |
| RATELIMITS_BLACKLIST_TOTAL_REQUESTS | | By default client names in the blacklist will be subject to 0 requests per hours. |
| RATELIMITS_BLACKLIST_EVERY_SECONDS | | By default client names in the blacklist will be subject to 0 requests per hours |
| RATELIMITS_BLACKLIST | example1.blacklist,example2.blacklist | Comma separated list of the clients which are in the blacklistpool. |
## Author
John Crepezzi <john.crepezzi@gmail.com>

19
docker-compose.yaml Normal file
View File

@ -0,0 +1,19 @@
version: '3.0'
services:
haste-server:
build: .
networks:
- db-network
environment:
- STORAGE_TYPE=memcached
- STORAGE_HOST=memcached
- STORAGE_PORT=11211
ports:
- 7777:7777
memcached:
image: memcached:latest
networks:
- db-network
networks:
db-network:

108
docker-entrypoint.js Normal file
View File

@ -0,0 +1,108 @@
const {
HOST,
PORT,
KEY_LENGTH,
MAX_LENGTH,
STATIC_MAX_AGE,
RECOMPRESS_STATIC_ASSETS,
STORAGE_TYPE,
STORAGE_HOST,
STORAGE_PORT,
STORAGE_EXPIRE_SECONDS,
STORAGE_DB,
STORAGE_AWS_BUCKET,
STORAGE_AWS_REGION,
STORAGE_PASSWORD,
STORAGE_USERNAME,
STORAGE_FILEPATH,
LOGGING_LEVEL,
LOGGING_TYPE,
LOGGING_COLORIZE,
KEYGENERATOR_TYPE,
KEY_GENERATOR_KEYSPACE,
RATE_LIMITS_NORMAL_TOTAL_REQUESTS,
RATE_LIMITS_NORMAL_EVERY_MILLISECONDS,
RATE_LIMITS_WHITELIST_TOTAL_REQUESTS,
RATE_LIMITS_WHITELIST_EVERY_MILLISECONDS,
RATE_LIMITS_WHITELIST,
RATE_LIMITS_BLACKLIST_TOTAL_REQUESTS,
RATE_LIMITS_BLACKLIST_EVERY_MILLISECONDS,
RATE_LIMITS_BLACKLIST,
DOCUMENTS,
} = process.env;
const config = {
host: HOST,
port: PORT,
keyLength: KEY_LENGTH,
maxLength: MAX_LENGTH,
staticMaxAge: STATIC_MAX_AGE,
recompressStaticAssets: RECOMPRESS_STATIC_ASSETS,
logging: [
{
level: LOGGING_LEVEL,
type: LOGGING_TYPE,
colorize: LOGGING_COLORIZE,
},
],
keyGenerator: {
type: KEYGENERATOR_TYPE,
keyspace: KEY_GENERATOR_KEYSPACE,
},
rateLimits: {
whitelist: RATE_LIMITS_WHITELIST ? RATE_LIMITS_WHITELIST.split(",") : [],
blacklist: RATE_LIMITS_BLACKLIST ? RATE_LIMITS_BLACKLIST.split(",") : [],
categories: {
normal: {
totalRequests: RATE_LIMITS_NORMAL_TOTAL_REQUESTS,
every: RATE_LIMITS_NORMAL_EVERY_MILLISECONDS,
},
whitelist:
RATE_LIMITS_WHITELIST_EVERY_MILLISECONDS ||
RATE_LIMITS_WHITELIST_TOTAL_REQUESTS
? {
totalRequests: RATE_LIMITS_WHITELIST_TOTAL_REQUESTS,
every: RATE_LIMITS_WHITELIST_EVERY_MILLISECONDS,
}
: null,
blacklist:
RATE_LIMITS_BLACKLIST_EVERY_MILLISECONDS ||
RATE_LIMITS_BLACKLIST_TOTAL_REQUESTS
? {
totalRequests: RATE_LIMITS_WHITELIST_TOTAL_REQUESTS,
every: RATE_LIMITS_BLACKLIST_EVERY_MILLISECONDS,
}
: null,
},
},
storage: {
type: STORAGE_TYPE,
host: STORAGE_HOST,
port: STORAGE_PORT,
expire: STORAGE_EXPIRE_SECONDS,
bucket: STORAGE_AWS_BUCKET,
region: STORAGE_AWS_REGION,
connectionUrl: `postgres://${STORAGE_USERNAME}:${STORAGE_PASSWORD}@${STORAGE_HOST}:${STORAGE_PORT}/${STORAGE_DB}`,
db: STORAGE_DB,
user: STORAGE_USERNAME,
password: STORAGE_PASSWORD,
path: STORAGE_FILEPATH,
},
documents: DOCUMENTS
? DOCUMENTS.split(",").reduce((acc, item) => {
const keyAndValueArray = item.replace(/\s/g, "").split("=");
return { ...acc, [keyAndValueArray[0]]: keyAndValueArray[1] };
}, {})
: null,
};
console.log(JSON.stringify(config));

9
docker-entrypoint.sh Normal file
View File

@ -0,0 +1,9 @@
#!/bin/bash
# We use this file to translate environmental variables to .env files used by the application
set -e
node ./docker-entrypoint.js > ./config.js
exec "$@"

View File

@ -16,33 +16,55 @@ var DocumentHandler = function(options) {
DocumentHandler.defaultKeyLength = 10;
// Handle retrieving a document
DocumentHandler.prototype.handleGet = function(key, response, skipExpire) {
DocumentHandler.prototype.handleGet = function(request, response, config) {
const key = request.params.id.split('.')[0];
const skipExpire = !!config.documents[key];
this.store.get(key, function(ret) {
if (ret) {
winston.verbose('retrieved document', { key: key });
response.writeHead(200, { 'content-type': 'application/json' });
response.end(JSON.stringify({ data: ret, key: key }));
if (request.method === 'HEAD') {
response.end();
} else {
response.end(JSON.stringify({ data: ret, key: key }));
}
}
else {
winston.warn('document not found', { key: key });
response.writeHead(404, { 'content-type': 'application/json' });
response.end(JSON.stringify({ message: 'Document not found.' }));
if (request.method === 'HEAD') {
response.end();
} else {
response.end(JSON.stringify({ message: 'Document not found.' }));
}
}
}, skipExpire);
};
// Handle retrieving the raw version of a document
DocumentHandler.prototype.handleRawGet = function(key, response, skipExpire) {
DocumentHandler.prototype.handleRawGet = function(request, response, config) {
const key = request.params.id.split('.')[0];
const skipExpire = !!config.documents[key];
this.store.get(key, function(ret) {
if (ret) {
winston.verbose('retrieved raw document', { key: key });
response.writeHead(200, { 'content-type': 'text/plain; charset=UTF-8' });
response.end(ret);
if (request.method === 'HEAD') {
response.end();
} else {
response.end(ret);
}
}
else {
winston.warn('raw document not found', { key: key });
response.writeHead(404, { 'content-type': 'application/json' });
response.end(JSON.stringify({ message: 'Document not found.' }));
if (request.method === 'HEAD') {
response.end();
} else {
response.end(JSON.stringify({ message: 'Document not found.' }));
}
}
}, skipExpire);
};

View File

@ -0,0 +1,56 @@
/*global require,module,process*/
var AWS = require('aws-sdk');
var winston = require('winston');
var AmazonS3DocumentStore = function(options) {
this.expire = options.expire;
this.bucket = options.bucket;
this.client = new AWS.S3({region: options.region});
};
AmazonS3DocumentStore.prototype.get = function(key, callback, skipExpire) {
var _this = this;
var req = {
Bucket: _this.bucket,
Key: key
};
_this.client.getObject(req, function(err, data) {
if(err) {
callback(false);
}
else {
callback(data.Body.toString('utf-8'));
if (_this.expire && !skipExpire) {
winston.warn('amazon s3 store cannot set expirations on keys');
}
}
});
}
AmazonS3DocumentStore.prototype.set = function(key, data, callback, skipExpire) {
var _this = this;
var req = {
Bucket: _this.bucket,
Key: key,
Body: data,
ContentType: 'text/plain'
};
_this.client.putObject(req, function(err, data) {
if (err) {
callback(false);
}
else {
callback(true);
if (_this.expire && !skipExpire) {
winston.warn('amazon s3 store cannot set expirations on keys');
}
}
});
}
module.exports = AmazonS3DocumentStore;

View File

@ -0,0 +1,89 @@
/*global require,module,process*/
const Datastore = require('@google-cloud/datastore');
const winston = require('winston');
class GoogleDatastoreDocumentStore {
// Create a new store with options
constructor(options) {
this.kind = "Haste";
this.expire = options.expire;
this.datastore = new Datastore();
}
// Save file in a key
set(key, data, callback, skipExpire) {
var expireTime = (skipExpire || this.expire === undefined) ? null : new Date(Date.now() + this.expire * 1000);
var taskKey = this.datastore.key([this.kind, key])
var task = {
key: taskKey,
data: [
{
name: 'value',
value: data,
excludeFromIndexes: true
},
{
name: 'expiration',
value: expireTime
}
]
};
this.datastore.insert(task).then(() => {
callback(true);
})
.catch(err => {
callback(false);
});
}
// Get a file from a key
get(key, callback, skipExpire) {
var taskKey = this.datastore.key([this.kind, key])
this.datastore.get(taskKey).then((entity) => {
if (skipExpire || entity[0]["expiration"] == null) {
callback(entity[0]["value"]);
}
else {
// check for expiry
if (entity[0]["expiration"] < new Date()) {
winston.info("document expired", {key: key, expiration: entity[0]["expiration"], check: new Date(null)});
callback(false);
}
else {
// update expiry
var task = {
key: taskKey,
data: [
{
name: 'value',
value: entity[0]["value"],
excludeFromIndexes: true
},
{
name: 'expiration',
value: new Date(Date.now() + this.expire * 1000)
}
]
};
this.datastore.update(task).then(() => {
})
.catch(err => {
winston.error("failed to update expiration", {error: err});
});
callback(entity[0]["value"]);
}
}
})
.catch(err => {
winston.error("Error retrieving value from Google Datastore", {error: err});
callback(false);
});
}
}
module.exports = GoogleDatastoreDocumentStore;

View File

@ -26,7 +26,7 @@ class MemcachedDocumentStore {
// Save file in a key
set(key, data, callback, skipExpire) {
this.client.set(key, data, skipExpire ? 0 : this.expire, (error) => {
this.client.set(key, data, skipExpire ? 0 : this.expire || 0, (error) => {
callback(!error);
});
}
@ -34,10 +34,12 @@ class MemcachedDocumentStore {
// Get a file from a key
get(key, callback, skipExpire) {
this.client.get(key, (error, data) => {
callback(error ? false : data);
const value = error ? false : data;
callback(value);
// Update the key so that the expiration is pushed forward
if (!skipExpire) {
if (value && !skipExpire) {
this.set(key, data, (updateSucceeded) => {
if (!updateSucceeded) {
winston.error('failed to update expiration on GET', {key});

View File

@ -0,0 +1,88 @@
var MongoClient = require('mongodb').MongoClient,
winston = require('winston');
var MongoDocumentStore = function (options) {
this.expire = options.expire;
this.connectionUrl = process.env.DATABASE_URl || options.connectionUrl;
};
MongoDocumentStore.prototype.set = function (key, data, callback, skipExpire) {
var now = Math.floor(new Date().getTime() / 1000),
that = this;
this.safeConnect(function (err, db) {
if (err)
return callback(false);
db.collection('entries').update({
'entry_id': key,
$or: [
{ expiration: -1 },
{ expiration: { $gt: now } }
]
}, {
'entry_id': key,
'value': data,
'expiration': that.expire && !skipExpire ? that.expire + now : -1
}, {
upsert: true
}, function (err, existing) {
if (err) {
winston.error('error persisting value to mongodb', { error: err });
return callback(false);
}
callback(true);
});
});
};
MongoDocumentStore.prototype.get = function (key, callback, skipExpire) {
var now = Math.floor(new Date().getTime() / 1000),
that = this;
this.safeConnect(function (err, db) {
if (err)
return callback(false);
db.collection('entries').findOne({
'entry_id': key,
$or: [
{ expiration: -1 },
{ expiration: { $gt: now } }
]
}, function (err, entry) {
if (err) {
winston.error('error persisting value to mongodb', { error: err });
return callback(false);
}
callback(entry === null ? false : entry.value);
if (entry !== null && entry.expiration !== -1 && that.expire && !skipExpire) {
db.collection('entries').update({
'entry_id': key
}, {
$set: {
'expiration': that.expire + now
}
}, function (err, result) { });
}
});
});
};
MongoDocumentStore.prototype.safeConnect = function (callback) {
MongoClient.connect(this.connectionUrl, function (err, db) {
if (err) {
winston.error('error connecting to mongodb', { error: err });
callback(err);
} else {
callback(undefined, db);
}
});
};
module.exports = MongoDocumentStore;

View File

@ -1,14 +1,16 @@
/*global require,module,process*/
var postgres = require('pg');
var winston = require('winston');
const {Pool} = require('pg');
// create table entries (id serial primary key, key varchar(255) not null, value text not null, expiration int, unique(key));
// A postgres document store
var PostgresDocumentStore = function (options) {
this.expireJS = options.expire;
this.connectionUrl = process.env.DATABASE_URL || options.connectionUrl;
const connectionString = process.env.DATABASE_URL || options.connectionUrl;
this.pool = new Pool({connectionString});
};
PostgresDocumentStore.prototype = {
@ -64,16 +66,15 @@ PostgresDocumentStore.prototype = {
// A connection wrapper
safeConnect: function (callback) {
postgres.connect(this.connectionUrl, function (err, client, done) {
if (err) {
winston.error('error connecting to postgres', { error: err });
callback(err);
this.pool.connect((error, client, done) => {
if (error) {
winston.error('error connecting to postgres', {error});
callback(error);
} else {
callback(undefined, client, done);
}
});
}
};
module.exports = PostgresDocumentStore;

1558
package-lock.json generated

File diff suppressed because it is too large Load Diff

View File

@ -14,25 +14,21 @@
},
"main": "haste",
"dependencies": {
"busboy": "0.2.4",
"connect": "^3.7.0",
"connect-ratelimit": "0.0.7",
"connect-route": "0.1.5",
"connect": "3.4.1",
"st": "1.1.0",
"winston": "0.6.2",
"redis-url": "0.1.0",
"pg": "^8.0.0",
"redis": "0.8.1",
"redis-url": "0.1.0",
"st": "^2.0.0",
"uglify-js": "3.1.6",
"busboy": "0.2.4",
"pg": "4.1.1"
"winston": "^2.0.0"
},
"devDependencies": {
"mocha": "^4.0.1"
"mocha": "^8.1.3"
},
"bundledDependencies": [],
"engines": {
"node": "8.1.4",
"npm": "5.2.0"
},
"bin": {
"haste-server": "./server.js"
},

View File

@ -33,10 +33,7 @@
},
"storage": {
"type": "memcached",
"host": "127.0.0.1",
"port": 11211,
"expire": 2592000
"type": "file"
},
"documents": {

View File

@ -108,9 +108,7 @@ if (config.rateLimits) {
}
function raw(request, response) {
var key = request.params.id.split('.')[0];
var skipExpire = !!config.documents[key];
return documentHandler.handleRawGet(key, response, skipExpire);
return documentHandler.handleRawGet(request, response, config);
}
// first look at API calls
@ -118,15 +116,21 @@ app.use(route(function(router) {
// get raw documents - support getting with extension
router.get('/raw/:id', raw);
router.get('/:id/raw', raw);
router.head('/raw/:id', raw);
router.head('/:id/raw', raw);
// add documents
router.post('/documents', function(request, response) {
return documentHandler.handlePost(request, response);
});
// get documents
router.get('/documents/:id', function(request, response) {
var key = request.params.id.split('.')[0];
var skipExpire = !!config.documents[key];
return documentHandler.handleGet(key, response, skipExpire);
return documentHandler.handleGet(request, response, config);
});
router.head('/documents/:id', function(request, response) {
return documentHandler.handleGet(request, response, config);
});
}));

View File

@ -17,6 +17,8 @@ textarea {
outline: none;
resize: none;
font-size: 13px;
margin-top: 0;
margin-bottom: 0;
}
/* the line numbers */
@ -31,6 +33,7 @@ textarea {
font-size: 13px;
font-family: monospace;
text-align: right;
user-select: none;
}
/* code box when locked */
@ -123,6 +126,7 @@ a.logo:hover {
font-size: 12px;
line-height: 14px;
padding: 10px 15px;
user-select: none;
}
#box3 .label {

View File

@ -64,7 +64,7 @@ haste_document.prototype.save = function(data, callback) {
type: 'post',
data: data,
dataType: 'json',
contentType: 'application/json; charset=utf-8',
contentType: 'text/plain; charset=utf-8',
success: function(res) {
_this.locked = true;
_this.key = res.key;
@ -170,8 +170,7 @@ haste.extensionMap = {
lua: 'lua', pas: 'delphi', java: 'java', cpp: 'cpp', cc: 'cpp', m: 'objectivec',
vala: 'vala', sql: 'sql', sm: 'smalltalk', lisp: 'lisp', ini: 'ini',
diff: 'diff', bash: 'bash', sh: 'bash', tex: 'tex', erl: 'erlang', hs: 'haskell',
md: 'markdown', txt: '', coffee: 'coffee', json: 'javascript',
swift: 'swift'
md: 'markdown', txt: '', coffee: 'coffee', swift: 'swift'
};
// Look up the extension preferred for a type

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long