Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
civitaspo committed Jul 7, 2015
1 parent b70fb7c commit 5572639
Showing 1 changed file with 28 additions and 7 deletions.
35 changes: 28 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,26 +1,40 @@
# Hdfs output plugin for Embulk

TODO: Write short description here and build.gradle file.
A File Output Plugin for Embulk to write HDFS.

## Overview

* **Plugin type**: output
* **Plugin type**: file output
* **Load all or nothing**: no
* **Resume supported**: no
* **Cleanup supported**: yes
* **Cleanup supported**: no

## Configuration

- **property1**: description (string, required)
- **property2**: description (integer, default: default-value)
- **config_files** list of paths to Hadoop's configuration files (array of strings, default: `[]`)
- **config** overwrites configuration parameters (hash, default: `{}`)
- **output_path** the path finally stored files. (string, default: `"/tmp/embulk.output.hdfs_output.%Y%m%d_%s"`)
- **working_path** the path temporary stored files. (string, default: `"/tmp/embulk.working.hdfs_output.%Y%m%d_%s"`)

## Example

```yaml
out:
type: hdfs
property1: example1
property2: example2
config_files:
- /etc/hadoop/conf/core-site.xml
- /etc/hadoop/conf/hdfs-site.xml
- /etc/hadoop/conf/mapred-site.xml
- /etc/hadoop/conf/yarn-site.xml
config:
fs.defaultFS: 'hdfs://hdp-nn1:8020'
dfs.replication: 1
mapreduce.client.submit.file.replication: 1
fs.hdfs.impl: 'org.apache.hadoop.hdfs.DistributedFileSystem'
fs.file.impl: 'org.apache.hadoop.fs.LocalFileSystem'
formatter:
type: csv
encoding: UTF-8
```
Expand All @@ -29,3 +43,10 @@ out:
```
$ ./gradlew gem
```

## Development

```
$ ./gradlew classpath
$ bundle exec embulk run -I lib example.yml
```

0 comments on commit 5572639

Please sign in to comment.