Changes in version 1.1.3 (2026-03-24) - Upkeep with Rcpp v1.1.1. Changes in version 1.1.2 (2025-02-16) - Bumped minimum R version to 4.2.0. - Refactored tagger_impl to improve performance of tokenize(split = TRUE). Changes in version 1.1.1 (2024-07-06) - tokenize now warns rather than throws an error when an invalid input is given during partial parsing. With this change, tokenize is no longer entirely aborted even if an invalid string is given. Parsing of those strings is simply skipped. Changes in version 1.1.0 (2024-02-17) - Corrected probabilistic IDF calculation by global_idf3. - Refactored bind_tf_idf2. - Breaking Change: Changed behavior when norm=TRUE. Cosine nomalization is now performed on tf_idf values as in the RMeCab package. - Added tf="itf" and idf="df" options. Changes in version 1.0.1 (2023-12-02) - Added wrappers around dictionary compiler of MeCab. Changes in version 0.9.5 (2023-07-09) - Removed audubon dependency for maintainability. - pack now preserves doc_id type when it's factor. Changes in version 0.9.4 (2023-06-03) - Updated Makevars for Unix alikes. Users can now use a file specified by the MECABRC environment variable or ~/.mecabrc to set up dictionaries. Changes in version 0.9.3 (2023-04-20) - Removed unnecessary C++ files. Changes in version 0.9.2 (2023-04-12) - Prepare for CRAN release. Changes in version 0.8.1 - For performance, tokenize now skips resetting the output encodings to UTF-8. Changes in version 0.8.0 - Breaking Change: Changed numbering style of 'sentence_id' when split is FALSE. - Added grain_size argument to tokenize. - Added new bind_lr function. Changes in version 0.7.4 - Use RcppParallel::parallelFor instead of tbb::parallel_for. There are no user's visible changes. Changes in version 0.7.1 - Fix documentations. There are no visible changes. Changes in version 0.7.0 - tokenize can now accept a character vector in addition to a data.frame like object. - gbs_tokenize is now deprecated. Please use the tokenize function instead. Changes in version 0.6.4 - Refactored is_blank. Changes in version 0.6.3 - Added the partial argument to gbs_tokenize and tokenize. This argument controls the partial parsing mode, which forces to extract given chunks of sentences when activated. Changes in version 0.6.2 - More friendly errors are returned when invalid dictionary path was provided. - Added new posDebugRcpp function. Changes in version 0.6.1 - Revert some missing examples. Changes in version 0.6.0 - Functions added in version '0.5.1' was moved to 'audubon' package (>= 0.4.0). Changes in version 0.5.1 - Added some new functions. - bind_tf_idf2 can calculate and bind the term frequency, inverse document frequency, and tf-idf of the tidy text dataset. - collapse_tokens, mute_tokens, and lexical_density can be used for handling a tidy text dataset of tokens. Changes in version 0.5.0 - gibasa now includes the MeCab source, so that users do not need to pre-install the MeCab library when building and installing the package (to use tokenize, it still requires MeCab and its dictionaries installed and available). Changes in version 0.4.1 - tokenize now preserves the original order of docid_field. Changes in version 0.4.0 - Added bind_tf_idf2 function and is_blank function. Changes in version 0.3.1 - Updated dependencies. Changes in version 0.3.0 - Changed build process on Windows. - Added a vignette. Changes in version 0.2.1 - prettify now can extract columns only specified by col_select. Changes in version 0.2.0 - Added a NEWS.md file to track changes to the package. - tokenize now takes a data.frame as its first argument, returns a data.frame only. The former function that gets character vector and returns a data.frame or named list was renamed as gbs_tokenize.