Archive-name: ai-faq/neural-nets/part4

Last-modified: 1999-11-21


Maintainer: (Warren S. Sarle)

Neural Network FAQ, part 4 of 7: Books, data, etc. Copyright 1997, 1998, 1999 by Warren S. Sarle, Cary, NC, USA. Reviews provided by other authors as cited below are copyrighted by those authors, who by submitting the reviews for the FAQ give permission for the review to be reproduced as part of the FAQ in any of the ways specified in part 1 of the FAQ.

This is part 4 (of 7) of a monthly posting to the Usenet newsgroup See the part 1 of this posting for full information what it is all about.

========== Questions ==========

Part 1: Introduction
Part 2: Learning
Part 3: Generalization
Part 4: Books, data, etc.
    Books and articles about Neural Networks?
      The Best
        The best of the best
        The best popular introduction to NNs
        The best introductory book for business executives
        The best elementary textbooks on practical use of NNs
        The best elementary textbook on using and programming NNs
        The best elementary textbooks on NN research
        The best intermediate textbooks on NNs
        The best advanced textbook covering NNs
        The best books on image and signal processing with NNs
        The best book on time-series forecasting with NNs
        The best books on reinforcement learning
        The best book on neurofuzzy systems
        The best comparison of NNs with other classification methods
      Books for the Beginner
      The Classics
      Introductory Journal Articles
      Not-quite-so-introductory Literature
      Books with Source Code (C, C++)
      The Worst
    Journals and magazines about Neural Networks?
    Conferences and Workshops on Neural Networks?
    Neural Network Associations?
    On-line and machine-readable information about NNs?
    How to benchmark learning methods?
    Databases for experimentation with NNs?
      UCI machine learning database
      UCI KDD Archive
      The neural-bench Benchmark collection
      Delve: Data for Evaluating Learning in Valid Experiments
      NIST special databases of the National Institute Of Standards And Technology:
      CEDAR CD-ROM 1: Database of Handwritten Cities, States, ZIP Codes, Digits, and Alphabetic Characters
      Time series archive
      USENIX Faces
      Linguistic Data Consortium
      Otago Speech Corpus
      Astronomical Time Series
      Miscellaneous Images
Part 5: Free software
Part 6: Commercial software
Part 7: Hardware and miscellaneous


Subject: Books and articles about Neural Networks?

Most books in print can be ordered online from Amazon's prices and search engine are good and their service is excellent.

Barnes & Noble seems to have a slightly larger stock of technical books than Amazon, but their search engine is inferior and their web response is slower. It's a good idea to check both Barnes & Noble and Amazon for availability and prices.

Bookpool at does not have as large a selection as Amazon or Barnes and Noble but they often offer exceptional discounts.

The neural networks reading group at the University of Illinois at Urbana-Champaign, the Artifical Neural Networks and Computational Brain Theory (ANNCBT) forum, has compiled a large number of book and paper reviews at, with an emphasis more on cognitive science rather than practical applications of NNs.

The Best

The best of the best

Bishop (1995) is clearly the single best book on artificial NNs. This book excels in organization and choice of material, and is a close runner-up to Ripley (1996) for accuracy. If you are new to the field, read it from cover to cover. If you have lots of experience with NNs, it's an excellent reference. If you don't know calculus, take a class. I hope a second edition comes out soon! For more information, see The best intermediate textbooks on NNs below.

If you have questions on feedforward nets that aren't answered by Bishop, try Masters (1993) or Reed and Marks (1999) for practical issues or Ripley (1996) for theortical issues, all of which are reviewed below.

The best popular introduction to NNs

Hinton, G.E. (1992), "How Neural Networks Learn from Experience", Scientific American, 267 (September), 144-151.
Author's Webpage: (official)
and (private)
Journal Webpage:
Additional Information: Unfortunately that article is not available there.

The best introductory book for business executives

Bigus, J.P. (1996), Data Mining with Neural Networks: Solving Business Problems--from Application Development to Decision Support, NY: McGraw-Hill, ISBN 0-07-005779-6, xvii+221 pages.
The stereotypical business executive (SBE) does not want to know how or why NNs work--he (SBEs are usually male) just wants to make money. The SBE may know what an average or percentage is, but he is deathly afraid of "statistics". He understands profit and loss but does not want to waste his time learning things involving complicated math, such as high-school algebra. For further information on the SBE, see the "Dilbert" comic strip.

Bigus has written an excellent introduction to NNs for the SBE. Bigus says (p. xv), "For business executives, managers, or computer professionals, this book provides a thorough introduction to neural network technology and the issues related to its application without getting bogged down in complex math or needless details. The reader will be able to identify common business problems that are amenable to the neural netwrk approach and will be sensitized to the issues that can affect successful completion of such applications." Bigus succeeds in explaining NNs at a practical, intuitive, and necessarily shallow level without formulas--just what the SBE needs. This book is far better than Caudill and Butler (1990), a popular but disastrous attempt to explain NNs without formulas.

Chapter 1 introduces data mining and data warehousing, and sketches some applications thereof. Chapter 2 is the semi-obligatory philosophico-historical discussion of AI and NNs and is well-written, although the SBE in a hurry may want to skip it. Chapter 3 is a very useful discussion of data preparation. Chapter 4 describes a variety of NNs and what they are good for. Chapter 5 goes into practical issues of training and testing NNs. Chapters 6 and 7 explain how to use the results from NNs. Chapter 8 discusses intelligent agents. Chapters 9 through 12 contain case histories of NN applications, including market segmentation, real-estate pricing, customer ranking, and sales forecasting.

Bigus provides generally sound advice. He briefly discusses overfitting and overtraining without going into much detail, although I think his advice on p. 57 to have at least two training cases for each connection is somewhat lenient, even for noise-free data. I do not understand his claim on pp. 73 and 170 that RBF networks have advantages over backprop networks for nonstationary inputs--perhaps he is using the word "nonstationary" in a sense different from the statistical meaning of the term. There are other things in the book that I would quibble with, but I did not find any of the flagrant errors that are common in other books on NN applications such as Swingler (1996).

The one serious drawback of this book is that it is more than one page long and may therefore tax the attention span of the SBE. But any SBE who succeeds in reading the entire book should learn enough to be able to hire a good NN expert to do the real work.

The best elementary textbooks on practical use of NNs

Smith, M. (1993). Neural Networks for Statistical Modeling, NY: Van Nostrand Reinhold.
Book Webpage (Publisher):
Additional Information: seems to be out of print.
Smith is not a statistician, but he has made an impressive effort to convey statistical fundamentals applied to neural networks. The book has entire brief chapters on overfitting and validation (early stopping and split-sample sample validation, which he incorrectly calls cross-validation), putting it a rung above most other introductions to NNs. There are also brief chapters on data preparation and diagnostic plots, topics usually ignored in elementary NN books. Only feedforward nets are covered in any detail.

Weiss, S.M. and Kulikowski, C.A. (1991), Computer Systems That Learn, Morgan Kaufmann. ISBN 1 55860 065 5.
Author's Webpage: Kulikowski:
Book Webpage (Publisher):
Additional Information: Information of Weiss, S.M. are not available.
Briefly covers at a very elementary level feedforward nets, linear and nearest-neighbor discriminant analysis, trees, and expert sytems, emphasizing practical applications. For a book at this level, it has an unusually good chapter on estimating generalization error, including bootstrapping.

Reed, R.D., and Marks, R.J, II (1999), Neural Smithing: Supervised Learning in Feedforward Artificial Neural Networks, Cambridge, MA: The MIT Press, ISBN 0-262-18190-8.
Author's Webpage: Marks:
Book Webpage (Publisher):
After you have read Smith (1993) or Weiss and Kulikowski (1991), consult Reed and Marks for practical details on training MLPs (other types of neural nets such as RBF networks are barely even mentioned). They provide extensive coverage of backprop and its variants, and they also survey conventional optimization algorithms. Their coverage of initialization methods, constructive networks, pruning, and regularization methods is unusually thorough. Unlike the vast majority of books on neural nets, this one has lots of really informative graphs. The chapter on generalization assessment is slightly weak, which is why you should read Smith (1993) or Weiss and Kulikowski (1991) first. Also, there is little information on data preparation, for which Smith (1993) and Masters (1993; see below) should be consulted. There is some elementary calculus, but not enough that it should scare off anybody. Many second-rate books treat neural nets as mysterious black boxes, but Reed and Marks open up the box and provide genuine insight into the way neural nets work.

One problem with the book is that the terms "validation set" and "test set" are used inconsistently.

The best elementary textbook on using and programming NNs

Masters, T. (1993), Practical Neural Network Recipes in C++, Academic Press, ISBN 0-12-479040-2, US $45 incl. disks.
Book Webpage (Publisher):
Masters has written three exceptionally good books on NNs (the two others are listed below). He combines generally sound practical advice with some basic statistical knowledge to produce a programming text that is far superior to the competition (see "The Worst" below). Not everyone likes his C++ code (the usual complaint is that the code is not sufficiently OO) but, unlike the code in some other books, Masters's code has been successfully compiled and run by some readers of Masters's books are well worth reading even for people who have no interest in programming.

Masters, T. (1995) Advanced Algorithms for Neural Networks: A C++ Sourcebook, NY: John Wiley and Sons, ISBN 0-471-10588-0
Book Webpage (Publisher):
Additional Information: One has to search.
Clear explanations of conjugate gradient and Levenberg-Marquardt optimization algorithms, simulated annealing, kernel regression (GRNN) and discriminant analysis (PNN), Gram-Charlier networks, dimensionality reduction, cross-validation, and bootstrapping.

The best elementary textbooks on NN research

Fausett, L. (1994), Fundamentals of Neural Networks: Architectures, Algorithms, and Applications, Englewood Cliffs, NJ: Prentice Hall, ISBN 0-13-334186-0. Also published as a Prentice Hall International Edition, ISBN 0-13-042250-9. Sample software (source code listings in C and Fortran) is included in an Instructor's Manual.

Book Webpage (Publisher):
Additional Information: The mentioned programs / additional support is not available.

Review by Ian Cresswell:

What a relief! As a broad introductory text this is without any doubt the best currently available in its area. It doesn't include source code of any kind (normally this is badly written and compiler specific). The algorithms for many different kinds of simple neural nets are presented in a clear step by step manner in plain English.

Equally, the mathematics is introduced in a relatively gentle manner. There are no unnecessary complications or diversions from the main theme.

The examples that are used to demonstrate the various algorithms are detailed but (perhaps necessarily) simple.

There are bad things that can be said about most books. There are only a small number of minor criticisms that can be made about this one. More space should have been given to backprop and its variants because of the practical importance of such methods. And while the author discusses early stopping in one paragraph, the treatment of generalization is skimpy compared to the books by Weiss and Kulikowski or Smith listed above.

If you're new to neural nets and you don't want to be swamped by bogus ideas, huge amounts of intimidating looking mathematics, a programming language that you don't know etc. etc. then this is the book for you.

In summary, this is the best starting point for the outsider and/or beginner... a truly excellent text.

Anderson, J.A. (1995), An Introduction to Neural Networks, Cambridge,MA: The MIT Press, ISBN 0-262-01144-1.
Author's Webpage:
Book Webpage (Publisher): or (hardback)
Additional Information: Programs and additional information can be found at:
Anderson provides an accessible introduction to the AI and neurophysiological sides of NN research, although the book is weak regarding practical aspects of using NNs. Recommended for classroom use if the instructor provides supplementary material on how to get good generalization.

The best intermediate textbooks on NNs

Bishop, C.M. (1995). Neural Networks for Pattern Recognition, Oxford: Oxford University Press. ISBN 0-19-853849-9 (hardback) or 0-19-853864-2 (paperback), xvii+482 pages.
Author's Webpage:
Book Webpage (Publisher):
This is definitely the best book on neural nets for practical applications for readers comfortable with calculus. Geoffrey Hinton writes in the foreword:
"Bishop is a leading researcher who has a deep understanding of the material and has gone to great lengths to organize it in a sequence that makes sense. He has wisely avoided the temptation to try to cover everything and has therefore omitted interesting topics like reinforcement learning, Hopfield networks, and Boltzmann machines in order to focus on the types of neural networks that are most widely used in practical applications. He assumes that the reader has the basic mathematical literacy required for an undergraduate science degree, and using these tools he explains everything from scratch. Before introducing the multilayer perceptron, for example, he lays a solid foundation of basic statistical concepts. So the crucial concept of overfitting is introduced using easily visualized examples of one-dimensional polynomials and only later applied to neural networks. An impressive aspect of this book is that it takes the reader all the way from the simplest linear models to the very latest Bayesian multilayer neural networks without ever requiring any great intellectual leaps."

Hertz, J., Krogh, A., and Palmer, R. (1991). Introduction to the Theory of Neural Computation. Redwood City, CA: Addison-Wesley, ISBN 0-201-50395-6 (hardbound) and 0-201-51560-1 (paperbound)
Book Webpage (Publisher):
This is an excellent classic work on neural nets from the perspective of physics. Comments from readers of "My first impression is that this one is by far the best book on the topic. And it's below $30 for the paperback."; "Well written, theoretical (but not overwhelming)"; It provides a good balance of model development, computational algorithms, and applications. The mathematical derivations are especially well done"; "Nice mathematical analysis on the mechanism of different learning algorithms"; "It is NOT for mathematical beginner. If you don't have a good grasp of higher level math, this book can be really tough to get through."

The best advanced textbook covering NNs

Ripley, B.D. (1996) Pattern Recognition and Neural Networks, Cambridge: Cambridge University Press, ISBN 0-521-46086-7 (hardback), xii+403 pages.
Author's Webpage:
Book Webpage (Publisher):
Additional Information: The Webpage includes errata and additional information, which hasn't been available at publishing time, for this book.
Brian Ripley's new book is an excellent sequel to Bishop (1995). Ripley starts up where Bishop left off, with Bayesian inference and statistical decision theory, and then covers some of the same material on NNs as Bishop but at a higher mathematical level. Ripley also covers a variety of methods that are not discussed, or discussed only briefly, by Bishop, such as tree-based methods and belief networks. While Ripley is best appreciated by people with a background in mathematical statistics, the numerous realistic examples in his book will be of interest even to beginners in neural nets.

Devroye, L., Gy\"orfi, L., and Lugosi, G. (1996), A Probabilistic Theory of Pattern Recognition, NY: Springer, ISBN 0-387-94618-7, vii+636 pages.
This book has relatively little material explicitly about neural nets, but what it has is very interesting and much of it is not found in other texts. The emphasis is on statistical proofs of universal consistency for a wide variety of methods, including histograms, (k) nearest neighbors, kernels (PNN), trees, generalized linear discriminants, MLPs, and RBF networks. There is also considerable material on validation and cross-validation. The authors say, "We did not scar the pages with backbreaking simulations or quick-and-dirty engineering solutions" (p. 7). The formula-to-text ratio is high, but the writing is quite clear, and anyone who has had a year or two of mathematical statistics should be able to follow the exposition.

The best books on image and signal processing with NNs

Masters, T. (1994), Signal and Image Processing with Neural Networks: A C++ Sourcebook, NY: Wiley.
Book Webpage (Publisher):
Additional Information: One has to search.

Cichocki, A. and Unbehauen, R. (1993). Neural Networks for Optimization and Signal Processing. NY: John Wiley & Sons, ISBN 0-471-930105 (hardbound), 526 pages, $57.95.
Book Webpage (Publisher):
Additional Information: One has to search.
Comments from readers of"Partly a textbook and partly a research monograph; introduces the basic concepts, techniques, and models related to neural networks and optimization, excluding rigorous mathematical details. Accessible to a wide readership with a differential calculus background. The main coverage of the book is on recurrent neural networks with continuous state variables. The book title would be more appropriate without mentioning signal processing. Well edited, good illustrations."

The best book on time-series forecasting with NNs

Weigend, A.S. and Gershenfeld, N.A., eds. (1994) Time Series Prediction: Forecasting the Future and Understanding the Past, Reading, MA: Addison-Wesley. Book Webpage (Publisher):

The best books on reinforcement learning

Sutton, R.S., and Barto, A.G. (1998), Reinforcement Learning: An Introduction, The MIT Press, ISBN: 0-262193-98-1.
Author's Webpage: and
Book Webpage (Publisher):
Additional Information:

Bertsekas, D. P. and Tsitsiklis, J. N. (1996), Neuro-Dynamic Programming, Belmont, MA: Athena Scientific, ISBN 1-886529-10-8.
Author's Webpage: and
Book Webpage (Publisher):

The best books on neurofuzzy systems

Brown, M., and Harris, C. (1994), Neurofuzzy Adaptive Modelling and Control, NY: Prentice Hall.
Author's Webpage:
Book Webpage (Publisher):
Additional Information: Additional page at: and an abstract can be found at:
Brown and Harris rely on the fundamental insight that that a fuzzy system is a nonlinear mapping from an input space to an output space that can be parameterized in various ways and therefore can be adapted to data using the usual neural training methods (see "What is backprop?") or conventional numerical optimization algorithms (see "What are conjugate gradients, Levenberg-Marquardt, etc.?"). Their approach makes clear the intimate connections between fuzzy systems, neural networks, and statistical methods such as B-spline regression.

Kosko, B. (1997), Fuzzy Engineering, Upper Saddle River, NJ: Prentice Hall.
Kosko's new book is a big improvement over his older neurofuzzy book and makes an excellent sequel to Brown and Harris (1994).

The best comparison of NNs with other classification methods

Michie, D., Spiegelhalter, D.J. and Taylor, C.C. (1994), Machine Learning, Neural and Statistical Classification, Ellis Horwood. Author's Webpage: Donald Michie:
Additional Information: This book is out of print but available online at

Books for the Beginner

Aleksander, I. and Morton, H. (1990). An Introduction to Neural Computing. Chapman and Hall. (ISBN 0-412-37780-2).
Book Webpage (Publisher):
Additional Information: Seems to be out of print.
Comments from readers of "This book seems to be intended for the first year of university education."

Beale, R. and Jackson, T. (1990). Neural Computing, an Introduction. Adam Hilger, IOP Publishing Ltd : Bristol. (ISBN 0-85274-262-2).
Comments from readers of "It's clearly written. Lots of hints as to how to get the adaptive models covered to work (not always well explained in the original sources). Consistent mathematical terminology. Covers perceptrons, error-backpropagation, Kohonen self-org model, Hopfield type models, ART, and associative memories."

Caudill, M. and Butler, C. (1990). Naturally Intelligent Systems. MIT Press: Cambridge, Massachusetts. (ISBN 0-262-03156-6).
Book Webpage (Publisher):
The authors try to translate mathematical formulas into English. The results are likely to disturb people who appreciate either mathematics or English. Have the authors never heard that "a picture is worth a thousand words"? What few diagrams they have (such as the one on p. 74) tend to be confusing. Their jargon is peculiar even by NN standards; for example, they refer to target values as "mentor inputs" (p. 66). The authors do not understand elementary properties of error functions and optimization algorithms. For example, in their discussion of the delta rule, the authors seem oblivious to the differences between batch and on-line training, and they attribute magical properties to the algorithm (p. 71):

[The on-line delta] rule always takes the most efficient route from the current position of the weight vector to the "ideal" position, based on the current input pattern. The delta rule not only minimizes the mean squared error, it does so in the most efficient fashion possible--quite an achievement for such a simple rule.
While the authors realize that backpropagation networks can suffer from local minima, they mistakenly think that counterpropagation has some kind of global optimization ability (p. 202):
Unlike the backpropagation network, a counterpropagation network cannot be fooled into finding a local minimum solution. This means that the network is guaranteed to find the correct response (or the nearest stored response) to an input, no matter what.
But even though they acknowledge the problem of local minima, the authors are ignorant of the importance of initial weight values (p. 186):
To teach our imaginary network something using backpropagation, we must start by setting all the adaptive weights on all the neurodes in it to random values. It won't matter what those values are, as long as they are not all the same and not equal to 1.
Like most introductory books, this one neglects the difficulties of getting good generalization--the authors simply declare (p. 8) that "A neural network is able to generalize"!

Chester, M. (1993). Neural Networks: A Tutorial, Englewood Cliffs, NJ: PTR Prentice Hall.
Book Webpage (Publisher):
Additional Information: Seems to be out of print.
Shallow, sometimes confused, especially with regard to Kohonen networks.

Dayhoff, J. E. (1990). Neural Network Architectures: An Introduction. Van Nostrand Reinhold: New York.
Comments from readers of "Like Wasserman's book, Dayhoff's book is also very easy to understand".

Freeman, James (1994). Simulating Neural Networks with Mathematica, Addison-Wesley, ISBN: 0-201-56629-X. Book Webpage (Publisher):
Additional Information: Sourcecode available under:
Helps the reader make his own NNs. The mathematica code for the programs in the book is also available through the internet: Send mail to or try on the World Wide Web.

Freeman, J.A. and Skapura, D.M. (1991). Neural Networks: Algorithms, Applications, and Programming Techniques, Reading, MA: Addison-Wesley.
Book Webpage (Publisher):
Additional Information: Seems to be out of print.
A good book for beginning programmers who want to learn how to write NN programs while avoiding any understanding of what NNs do or why they do it.

Gately, E. (1996). Neural Networks for Financial Forecasting. New York: John Wiley and Sons, Inc.
Book Webpage (Publisher):
Additional Information: One has to search.
Franco Insana comments:

* Decent book for the neural net beginner

* Very little devoted to statistical framework, although there 

    is some formulation of backprop theory

* Some food for thought

* Nothing here for those with any neural net experience

Hecht-Nielsen, R. (1990). Neurocomputing. Addison Wesley.
Book Webpage (Publisher):
Additional Information: Seems to be out of print.
Comments from readers of "A good book", "comprises a nice historical overview and a chapter about NN hardware. Well structured prose. Makes important concepts clear."

McClelland, J. L. and Rumelhart, D. E. (1988). Explorations in Parallel Distributed Processing: Computational Models of Cognition and Perception (software manual). The MIT Press.
Book Webpage (Publisher): (IBM version) and (Macintosh)
Comments from readers of "Written in a tutorial style, and includes 2 diskettes of NN simulation programs that can be compiled on MS-DOS or Unix (and they do too !)"; "The programs are pretty reasonable as an introduction to some of the things that NNs can do."; "There are *two* editions of this book. One comes with disks for the IBM PC, the other comes with disks for the Macintosh".

McCord Nelson, M. and Illingworth, W.T. (1990). A Practical Guide to Neural Nets. Addison-Wesley Publishing Company, Inc. (ISBN 0-201-52376-0).
Book Webpage (Publisher):
Lots of applications without technical details, lots of hype, lots of goofs, no formulas.

Muller, B., Reinhardt, J., Strickland, M. T. (1995). Neural Networks.:An Introduction (2nd ed.). Berlin, Heidelberg, New York: Springer-Verlag. ISBN 3-540-60207-0. (DOS 3.5" disk included.)
Book Webpage (Publisher):
Comments from readers of "The book was developed out of a course on neural-network models with computer demonstrations that was taught by the authors to Physics students. The book comes together with a PC-diskette. The book is divided into three parts: (1) Models of Neural Networks; describing several architectures and learing rules, including the mathematics. (2) Statistical Physiscs of Neural Networks; "hard-core" physics section developing formal theories of stochastic neural networks. (3) Computer Codes; explanation about the demonstration programs. First part gives a nice introduction into neural networks together with the formulas. Together with the demonstration programs a 'feel' for neural networks can be developed."

Orchard, G.A. & Phillips, W.A. (1991). Neural Computation: A Beginner's Guide. Lawrence Earlbaum Associates: London.
Comments from readers of "Short user-friendly introduction to the area, with a non-technical flavour. Apparently accompanies a software package, but I haven't seen that yet".

Rao, V.B & H.V. (1993). C++ Neural Networks and Fuzzy Logic. MIS:Press, ISBN 1-55828-298-x, US $45 incl. disks.
Covers a wider variety of networks than Masters (1993), Practical Neural Network Recipes in C++,, but lacks Masters's insight into practical issues of using NNs.

Wasserman, P. D. (1989). Neural Computing: Theory & Practice. Van Nostrand Reinhold: New York. (ISBN 0-442-20743-3)
Comments from readers of "Wasserman flatly enumerates some common architectures from an engineer's perspective ('how it works') without ever addressing the underlying fundamentals ('why it works') - important basic concepts such as clustering, principal components or gradient descent are not treated. It's also full of errors, and unhelpful diagrams drawn with what appears to be PCB board layout software from the '70s. For anyone who wants to do active research in the field I consider it quite inadequate"; "Okay, but too shallow"; "Quite easy to understand"; "The best bedtime reading for Neural Networks. I have given this book to numerous collegues who want to know NN basics, but who never plan to implement anything. An excellent book to give your manager."

The Classics

Kohonen, T. (1984). Self-organization and Associative Memory. Springer-Verlag: New York. (2nd Edition: 1988; 3rd edition: 1989).
Author's Webpage:
Book Webpage (Publisher):
Additional Information: Book is out of print.
Comments from readers of "The section on Pattern mathematics is excellent."

Rumelhart, D. E. and McClelland, J. L. (1986). Parallel Distributed Processing: Explorations in the Microstructure of Cognition (volumes 1 & 2). The MIT Press.
Author's Webpage:
Book Webpage (Publisher):
Comments from readers of "As a computer scientist I found the two Rumelhart and McClelland books really heavy going and definitely not the sort of thing to read if you are a beginner."; "It's quite readable, and affordable (about $65 for both volumes)."; "THE Connectionist bible".

Introductory Journal Articles

Hinton, G. E. (1989). Connectionist learning procedures. Artificial Intelligence, Vol. 40, pp. 185--234.
Author's Webpage: (official) and (private)
Journal Webpage (Publisher):
Comments from readers of "One of the better neural networks overview papers, although the distinction between network topology and learning algorithm is not always very clear. Could very well be used as an introduction to neural networks."

Knight, K. (1990). Connectionist, Ideas and Algorithms. Communications of the ACM. November 1990. Vol.33 nr.11, pp 59-74.
Comments from readers of"A good article, while it is for most people easy to find a copy of this journal."

Kohonen, T. (1988). An Introduction to Neural Computing. Neural Networks, vol. 1, no. 1. pp. 3-16.
Author's Webpage:
Journal Webpage (Publisher):
Additional Information: Article not available there.
Comments from readers of "A general review".

Rumelhart, D. E., Hinton, G. E. and Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, vol 323 (9 October), pp. 533-536.
Journal Webpage (Publisher):
Additional Information: Article not available there.
Comments from readers of "Gives a very good potted explanation of backprop NN's. It gives sufficient detail to write your own NN simulation."

Not-quite-so-introductory Literature

Anderson, J. A. and Rosenfeld, E. (Eds). (1988). Neurocomputing: Foundations of Research. The MIT Press: Cambridge, MA.
Author's Webpage:
Book Webpage (Publisher):
Comments from readers of "An expensive book, but excellent for reference. It is a collection of reprints of most of the major papers in the field."

Anderson, J. A., Pellionisz, A. and Rosenfeld, E. (Eds). (1990). Neurocomputing 2: Directions for Research. The MIT Press: Cambridge, MA.
Author's Webpage:
Book Webpage (Publisher):
Comments from readers of "The sequel to their well-known Neurocomputing book."

Bourlard, H.A., and Morgan, N. (1994), Connectionist Speech Recognition: A Hybrid Approach, Boston: Kluwer Academic Publishers.

Deco, G. and Obradovic, D. (1996), An Information-Theoretic Approach to Neural Computing, NY: Springer-Verlag.

Haykin, S. (1994). Neural Networks, a Comprehensive Foundation. Macmillan, New York, NY.
Comments from readers of "A very readable, well written intermediate text on NNs Perspective is primarily one of pattern recognition, estimation and signal processing. However, there are well-written chapters on neurodynamics and VLSI implementation. Though there is emphasis on formal mathematical models of NNs as universal approximators, statistical estimators, etc., there are also examples of NNs used in practical applications. The problem sets at the end of each chapter nicely complement the material. In the bibliography are over 1000 references."

Khanna, T. (1990). Foundations of Neural Networks. Addison-Wesley: New York.
Book Webpage (Publisher):
Comments from readers of "Not so bad (with a page of erroneous formulas (if I remember well), and #hidden layers isn't well described)."; "Khanna's intention in writing his book with math analysis should be commended but he made several mistakes in the math part".

Kung, S.Y. (1993). Digital Neural Networks, Prentice Hall, Englewood Cliffs, NJ.

Book Webpage (Publisher):
Levine, D. S. (1990). Introduction to Neural and Cognitive Modeling. Lawrence Erlbaum: Hillsdale, N.J.
Comments from readers of "Highly recommended".

Lippmann, R. P. (April 1987). An introduction to computing with neural nets. IEEE Acoustics, Speech, and Signal Processing Magazine. vol. 2, no. 4, pp 4-22.
Comments from readers of "Much acclaimed as an overview of neural networks, but rather inaccurate on several points. The categorization into binary and continuous- valued input neural networks is rather arbitrary, and may work confusing for the unexperienced reader. Not all networks discussed are of equal importance."

Maren, A., Harston, C. and Pap, R., (1990). Handbook of Neural Computing Applications. Academic Press. ISBN: 0-12-471260-6. (451 pages)
Comments from readers of "They cover a broad area"; "Introductory with suggested applications implementation".

Pao, Y. H. (1989). Adaptive Pattern Recognition and Neural Networks Addison-Wesley Publishing Company, Inc. (ISBN 0-201-12584-6)
Book Webpage (Publisher):
Comments from readers of "An excellent book that ties together classical approaches to pattern recognition with Neural Nets. Most other NN books do not even mention conventional approaches."

Refenes, A. (Ed.) (1995). Neural Networks in the Capital Markets. Chichester, England: John Wiley and Sons, Inc.
Book Webpage (Publisher):
Additional Information: One has to search.
Franco Insana comments:

* Not for the beginner

* Excellent introductory material presented by editor in first 5 

  chapters, which could be a valuable reference source for any 


* Very thought-provoking

* Mostly backprop-related

* Most contributors lay good statistical foundation

* Overall, a wealth of information and ideas, but the reader has to 

  sift through it all to come away with anything useful

Simpson, P. K. (1990). Artificial Neural Systems: Foundations, Paradigms, Applications and Implementations. Pergamon Press: New York.
Comments from readers of "Contains a very useful 37 page bibliography. A large number of paradigms are presented. On the negative side the book is very shallow. Best used as a complement to other books".

Wasserman, P.D. (1993). Advanced Methods in Neural Computing. Van Nostrand Reinhold: New York (ISBN: 0-442-00461-3).
Comments from readers of "Several neural network topics are discussed e.g. Probalistic Neural Networks, Backpropagation and beyond, neural control, Radial Basis Function Networks, Neural Engineering. Furthermore, several subjects related to neural networks are mentioned e.g. genetic algorithms, fuzzy logic, chaos. Just the functionality of these subjects is described; enough to get you started. Lots of references are given to more elaborate descriptions. Easy to read, no extensive mathematical background necessary."

Zeidenberg. M. (1990). Neural Networks in Artificial Intelligence. Ellis Horwood, Ltd., Chichester.
Comments from readers of "Gives the AI point of view".

Zornetzer, S. F., Davis, J. L. and Lau, C. (1990). An Introduction to Neural and Electronic Networks. Academic Press. (ISBN 0-12-781881-2)
Comments from readers of "Covers quite a broad range of topics (collection of articles/papers )."; "Provides a primer-like introduction and overview for a broad audience, and employs a strong interdisciplinary emphasis".

Zurada, Jacek M. (1992). Introduction To Artificial Neural Systems. Hardcover, 785 Pages, 317 Figures, ISBN 0-534-95460-X, 1992, PWS Publishing Company, Price: $56.75 (includes shipping, handling, and the ANS software diskette). Solutions Manual available.
Comments from readers of "Cohesive and comprehensive book on neural nets; as an engineering-oriented introduction, but also as a research foundation. Thorough exposition of fundamentals, theory and applications. Training and recall algorithms appear in boxes showing steps of algorithms, thus making programming of learning paradigms easy. Many illustrations and intuitive examples. Winner among NN textbooks at a senior UG/first year graduate level-[175 problems]." Contents: Intro, Fundamentals of Learning, Single-Layer & Multilayer Perceptron NN, Assoc. Memories, Self-organizing and Matching Nets, Applications, Implementations, Appendix)

Books with Source Code (C, C++)

Blum, Adam (1992), Neural Networks in C++, Wiley.

Review by Ian Cresswell. (For a review of the text, see "The Worst" below.)

Mr Blum has not only contributed a masterpiece of NN inaccuracy but also seems to lack a fundamental understanding of Object Orientation.

The excessive use of virtual methods (see page 32 for example), the inclusion of unnecessary 'friend' relationships (page 133) and a penchant for operator overloading (pick a page!) demonstrate inability in C++ and/or OO.

The introduction to OO that is provided trivialises the area and demonstrates a distinct lack of direction and/or understanding.

The public interfaces to classes are overspecified and the design relies upon the flawed neuron/layer/network model.

There is a notable disregard for any notion of a robust class hierarchy which is demonstrated by an almost total lack of concern for inheritance and associated reuse strategies.

The attempt to rationalise differing types of Neural Network into a single very shallow but wide class hierarchy is naive.

The general use of the 'float' data type would cause serious hassle if this software could possibly be extended to use some of the more sensitive variants of backprop on more difficult problems. It is a matter of great fortune that such software is unlikely to be reusable and will therefore, like all good dinosaurs, disappear with the passage of time.

The irony is that there is a card in the back of the book asking the unfortunate reader to part with a further $39.95 for a copy of the software (already included in print) on a 5.25" disk.

The author claims that his work provides an 'Object Oriented Framework ...'. This can best be put in his own terms (Page 137):

... garble(float noise) ...

Swingler, K. (1996), Applying Neural Networks: A Practical Guide, London: Academic Press.

Review by Ian Cresswell. (For a review of the text, see "The Worst" below.)

Before attempting to review the code associated with this book it should be clearly stated that it is supplied as an extra--almost as an afterthought. This may be a wise move.

Although not as bad as other (even commercial) implementations, the code provided lacks proper OO structure and is typical of C++ written in a C style.

Style criticisms include:

  1. The use of public data fields within classes (loss of encapsulation).
  2. Classes with no protected or private sections.
  3. Little or no use of inheritance and/or run-time polymorphism.
  4. Use of floats not doubles (a common mistake) to store values for connection weights.
  5. Overuse of classes and public methods. The network class has 59 methods in its public section.
  6. Lack of planning is evident for the construction of a class hierarchy.
This code is without doubt written by a rushed C programmer. Whilst it would require a C++ compiler to be successfully used, it lacks the tight (optimised) nature of good C and the high level of abstraction of good C++.

In a generous sense the code is free and the author doesn't claim any expertise in software engineering. It works in a limited sense but would be difficult to extend and/or reuse. It's fine for demonstration purposes in a stand-alone manner and for use with the book concerned.

If you're serious about nets you'll end up rewriting the whole lot (or getting something better).

The Worst

How not to use neural nets in any programming language

Blum, Adam (1992), Neural Networks in C++, NY: Wiley.

Welstead, Stephen T. (1994), Neural Network and Fuzzy Logic Applications in C/C++, NY: Wiley.

(For a review of Blum's source code, see "Books with Source Code" above.)

Both Blum and Welstead contribute to the dangerous myth that any idiot can use a neural net by dumping in whatever data are handy and letting it train for a few days. They both have little or no discussion of generalization, validation, and overfitting. Neither provides any valid advice on choosing the number of hidden nodes. If you have ever wondered where these stupid "rules of thumb" that pop up frequently come from, here's a source for one of them:

"A rule of thumb is for the size of this [hidden] layer to be somewhere between the input layer size ... and the output layer size ..." Blum, p. 60.
(John Lazzaro tells me he recently "reviewed a paper that cited this rule of thumb--and referenced this book! Needless to say, the final version of that paper didn't include the reference!")

Blum offers some profound advice on choosing inputs:

"The next step is to pick as many input factors as possible that might be related to [the target]."
Blum also shows a deep understanding of statistics:
"A statistical model is simply a more indirect way of learning correlations. With a neural net approach, we model the problem directly." p. 8.
Blum at least mentions some important issues, however simplistic his advice may be. Welstead just ignores them. What Welstead gives you is code--vast amounts of code. I have no idea how anyone could write that much code for a simple feedforward NN. Welstead's approach to validation, in his chapter on financial forecasting, is to reserve two cases for the validation set!

My comments apply only to the text of the above books. I have not examined or attempted to compile the code.

An impractical guide to neural nets

Swingler, K. (1996), Applying Neural Networks: A Practical Guide, London: Academic Press.

(For a review of the source code, see "Books with Source Code" above.)

This book has lots of good advice liberally sprinkled with errors, incorrect formulas, some bad advice, and some very serious mistakes. Experts will learn nothing, while beginners will be unable to separate the useful information from the dangerous. For example, there is a chapter on "Data encoding and re-coding" that would be very useful to beginners if it were accurate, but the formula for the standard deviation is wrong, and the description of the softmax function is of something entirely different than softmax (see What is a softmax activation function?). Even more dangerous is the statement on p. 28 that "Any pair of variables with high covariance are dependent, and one may be chosen to be discarded." Although high correlations can be used to identify redundant inputs, it is incorrect to use high covariances for this purpose, since a covariance can be high simply because one of the inputs has a high standard deviation.

The most ludicrous thing I've found in the book is the claim that Hecht-Neilsen used Kolmogorov's theorem to show that "you will never require more than twice the number of hidden units as you have inputs" (p. 53) in an MLP with one hidden layer. Actually, Hecht-Neilsen, says "the direct usefulness of this result is doubtful, because no constructive method for developing the [output activation] functions is known." Then Swingler implies that V. Kurkova (1991, "Kolmogorov's theorem is relevant," Neural Computation, 3, 617-622) confirmed this alleged upper bound on the number of hidden units, saying that, "Kurkova was able to restate Kolmogorov's theorem in terms of a set of sigmoidal functions." If Kolmogorov's theorem, or Hecht-Nielsen's adaptation of it, could be restated in terms of known sigmoid activation functions in the (single) hidden and output layers, then Swingler's alleged upper bound would be correct, but in fact no such restatement of Kolmogorov's theorem is possible, and Kurkova did not claim to prove any such restatement. Swingler omits the crucial details that Kurkova used two hidden layers, staircase-like activation functions (not ordinary sigmoidal functions such as the logistic) in the first hidden layer, and a potentially large number of units in the second hidden layer. Kurkova later estimated the number of units required for uniform approximation within an error epsilon as nm(m+1) in the first hidden layer and m^2(m+1)^n in the second hidden layer, where n is the number of inputs and m "depends on epsilon/||f|| as well as on the rate with which f increases distances." In other words, Kurkova says nothing to support Swinglers advice (repeated on p. 55), "Never choose h to be more than twice the number of input units." Furthermore, constructing a counter example to Swingler's advice is trivial: use one input and one output, where the output is the sine of the input, and the domain of the input extends over many cycles of the sine wave; it is obvious that many more than two hidden units are required. For some sound information on choosing the number of hidden units, see How many hidden units should I use?

Choosing the number of hidden units is one important aspect of getting good generalization, which is the most crucial issue in neural network training. There are many other considerations involved in getting good generalization, and Swingler makes several more mistakes in this area:

While Swingler has some knowldege of statistics, his expertise is not sufficient for him to detect that certain articles on neural nets are statistically nonsense. For example, on pp. 139-140 he uncritically reports a method that allegedly obtains error bars by doing a simple linear regression on the target vs. output scores. To a trained statistician, this method is obviously wrong (and, as usual in this book, the formula for variance given for this method on p. 150 is wrong). On p. 110, Swingler reports an article that attempts to apply bootstrapping to neural nets, but this article is also obviously wrong to anyone familiar with bootstrapping. While Swingler cannot be blamed entirely for accepting these articles at face value, such misinformation provides yet more hazards for beginners.

Swingler addresses many important practical issues, and often provides good practical advice. But the peculiar combination of much good advice with some extremely bad advice, a few examples of which are provided above, could easily seduce a beginner into thinking that the book as a whole is reliable. It is this danger that earns the book a place in "The Worst" list.

Bad science writing

Dewdney, A.K. (1997), Yes, We Have No Neutrons: An Eye-Opening Tour through the Twists and Turns of Bad Science, NY: Wiley.

This book, allegedly an expose of bad science, contains only one chapter of 19 pages on "the neural net debacle" (p. 97). Yet this chapter is so egregiously misleading that the book has earned a place on "The Worst" list. A detailed criticism of this chapter, along with some other sections of the book, can be found at Other chapters of the book are reviewed in the November, 1997, issue of Scientific American.


Subject: Journals and magazines about Neural Networks?

[to be added: comments on speed of reviewing and publishing,

              whether they accept TeX format or ASCII by e-mail, etc.]

A. Dedicated Neural Network Journals:

Title:   Neural Networks

Publish: Pergamon Press

Address: Pergamon Journals Inc., Fairview Park, Elmsford,

         New York 10523, USA and Pergamon Journals Ltd.

         Headington Hill Hall, Oxford OX3, 0BW, England

Freq.:   10 issues/year (vol. 1 in 1988)

Cost/Yr: Free with INNS or JNNS or ENNS membership ($45?),

         Individual $65, Institution $175

ISSN #:  0893-6080


Remark:  Official Journal of International Neural Network Society (INNS),

         European Neural Network Society (ENNS) and Japanese Neural

         Network Society (JNNS).

         Contains Original Contributions, Invited Review Articles, Letters

         to Editor, Book Reviews, Editorials, Announcements, Software Surveys.

Title:   Neural Computation

Publish: MIT Press

Address: MIT Press Journals, 55 Hayward Street Cambridge,

         MA 02142-9949, USA, Phone: (617) 253-2889

Freq.:   Quarterly (vol. 1 in 1989)

Cost/Yr: Individual $45, Institution $90, Students $35; Add $9 Outside USA

ISSN #:  0899-7667


Remark:  Combination of Reviews (10,000 words), Views (4,000 words)

         and Letters (2,000 words).  I have found this journal to be of

         outstanding quality.

         (Note: Remarks supplied by Mike Plonski "")

Title:   IEEE Transactions on Neural Networks

Publish: Institute of Electrical and Electronics Engineers (IEEE)

Address: IEEE Service Cemter, 445 Hoes Lane, P.O. Box 1331, Piscataway, NJ,

         08855-1331 USA. Tel: (201) 981-0060

Cost/Yr: $10 for Members belonging to participating IEEE societies

Freq.:   Quarterly (vol. 1 in March 1990)


Remark:  Devoted to the science and technology of neural networks

         which disclose significant  technical knowledge, exploratory

         developments and applications of neural networks from biology to

         software to hardware.  Emphasis is on artificial neural networks.

         Specific aspects include self organizing systems, neurobiological

         connections, network dynamics and architecture, speech recognition,

         electronic and photonic implementation, robotics and controls.

         Includes Letters concerning new research results.

         (Note: Remarks are from journal announcement)

Title:   IEEE Transactions on Evolutionary Computation

Publish: Institute of Electrical and Electronics Engineers (IEEE)

Address: IEEE Service Cemter, 445 Hoes Lane, P.O. Box 1331, Piscataway, NJ,

         08855-1331 USA. Tel: (201) 981-0060

Cost/Yr: $10 for Members belonging to participating IEEE societies

Freq.:   Quarterly (vol. 1 in May 1997)


Remark:  The IEEE Transactions on Evolutionary Computation will publish archival

         journal quality original papers in evolutionary computation and related

         areas, with particular emphasis on the practical application of the

         techniques to solving real problems in industry, medicine, and other

         disciplines.  Specific techniques include but are not limited to

         evolution strategies, evolutionary programming, genetic algorithms, and

         associated methods of genetic programming and classifier systems.  Papers

         emphasizing mathematical results should ideally seek to put these results

         in the context of algorithm design, however purely theoretical papers will

         be considered.  Other papers in the areas of cultural algorithms, artificial

         life, molecular computing, evolvable hardware, and the use of simulated

         evolution to gain a better understanding of naturally evolved systems are

         also encouraged.

         (Note: Remarks are from journal CFP)

Title:   International Journal of Neural Systems

Publish: World Scientific Publishing

Address: USA: World Scientific Publishing Co., 1060 Main Street, River Edge,

         NJ 07666. Tel: (201) 487 9655; Europe: World Scientific Publishing

         Co. Ltd., 57 Shelton Street, London WC2H 9HE, England.

         Tel: (0171) 836 0888; Asia: World Scientific Publishing Co. Pte. Ltd.,

         1022 Hougang Avenue 1 #05-3520, Singapore 1953, Rep. of Singapore

         Tel: 382 5663.

Freq.:   Quarterly (Vol. 1 in 1990)

Cost/Yr: Individual $122, Institution $255 (plus $15-$25 for postage)

ISSN #:  0129-0657 (IJNS)

Remark:  The International Journal of Neural Systems is a quarterly

         journal which covers information processing in natural

         and artificial neural systems. Contributions include research papers,

         reviews, and Letters to the Editor - communications under 3,000

         words in length, which are published within six months of receipt.

         Other contributions are typically published within nine months.

         The journal presents a fresh undogmatic attitude towards this

         multidisciplinary field and aims to be a forum for novel ideas and

         improved understanding of collective and cooperative phenomena with

         computational capabilities.

         Papers should be submitted to World Scientific's UK office. Once a

         paper is accepted for publication, authors are invited to e-mail

         the LaTeX source file of their paper in order to expedite publication.

Title:   International Journal of Neurocomputing

Publish: Elsevier Science Publishers, Journal Dept.; PO Box 211;

         1000 AE Amsterdam, The Netherlands

Freq.:   Quarterly (vol. 1 in 1989)


Title:   Neural Processing Letters

Publish: Kluwer Academic publishers

Address: P.O. Box 322, 3300 AH Dordrecht, The Netherlands

Freq:    6 issues/year (vol. 1 in 1994)

Cost/Yr: Individuals $198, Institution $400 (including postage)

ISSN #:  1370-4621


Remark:  The aim of the journal is to rapidly publish new ideas, original

         developments and work in progress.  Neural Processing Letters

         covers all aspects of the Artificial Neural Networks field.

         Publication delay is about 3 months.

Title:   Neural Network News

Publish: AIWeek Inc.

Address: Neural Network News, 2555 Cumberland Parkway, Suite 299,

         Atlanta, GA 30339 USA. Tel: (404) 434-2187

Freq.:   Monthly (beginning September 1989)

Cost/Yr: USA and Canada $249, Elsewhere $299

Remark:  Commercial Newsletter

Title:   Network: Computation in Neural Systems

Publish: IOP Publishing Ltd

Address: Europe: IOP Publishing Ltd, Techno House, Redcliffe Way, Bristol

         BS1 6NX, UK; IN USA: American Institute of Physics, Subscriber

         Services 500 Sunnyside Blvd., Woodbury, NY  11797-2999

Freq.:   Quarterly (1st issue 1990)

Cost/Yr: USA: $180,  Europe: 110 pounds

Remark:  Description: "a forum for integrating theoretical and experimental

         findings across relevant interdisciplinary boundaries."  Contents:

         Submitted articles reviewed by two technical referees  paper's

         interdisciplinary format and accessability."  Also Viewpoints and

         Reviews commissioned by the editors, abstracts (with reviews) of

         articles published in other journals, and book reviews.

         Comment: While the price discourages me (my comments are based

         upon a free sample copy), I think that the journal succeeds

         very well.  The highest density of interesting articles I

         have found in any journal.

         (Note: Remarks supplied by kehoe@csufres.CSUFresno.EDU)

Title:   Connection Science: Journal of Neural Computing,

         Artificial Intelligence and Cognitive Research

Publish: Carfax Publishing

Address: Europe: Carfax Publishing Company, PO Box 25, Abingdon, Oxfordshire

         OX14 3UE, UK.

         USA: Carfax Publishing Company, PO Box 2025, Dunnellon, Florida

         34430-2025, USA

         Australia: Carfax Publishing Company, Locked Bag 25, Deakin,

         ACT 2600, Australia

Freq.:   Quarterly (vol. 1 in 1989)

Cost/Yr: Personal rate:

         48 pounds (EC) 66 pounds (outside EC) US$118 (USA and Canada)

         Institutional rate:

         176 pounds (EC) 198 pounds (outside EC) US$340 (USA and Canada)

Title:   International Journal of Neural Networks

Publish: Learned Information

Freq.:   Quarterly (vol. 1 in 1989)

Cost/Yr: 90 pounds

ISSN #:  0954-9889

Remark:  The journal contains articles, a conference report (at least the

         issue I have), news and a calendar.

         (Note: remark provided by J.R.M. Smits "")

Title:   Sixth Generation Systems (formerly Neurocomputers)

Publish: Gallifrey Publishing

Address: Gallifrey Publishing, PO Box 155, Vicksburg, Michigan, 49097, USA

         Tel: (616) 649-3772, 649-3592 fax

Freq.    Monthly (1st issue January, 1987)

ISSN #:  0893-1585

Editor:  Derek F. Stubbs

Cost/Yr: $79 (USA, Canada), US$95 (elsewhere)

Remark:  Runs eight to 16 pages monthly. In 1995 will go to floppy disc-based

publishing with databases +, "the equivalent to 50 pages per issue are

planned." Often focuses on specific topics: e.g., August, 1994 contains two

articles: "Economics, Times Series and the Market," and "Finite Particle

Analysis - [part] II."  Stubbs also directs the company Advanced Forecasting

Technologies. (Remark by Ed Rosenfeld:

Title:   JNNS Newsletter (Newsletter of the Japan Neural Network Society)

Publish: The Japan Neural Network Society

Freq.:   Quarterly (vol. 1 in 1989)

Remark:  (IN JAPANESE LANGUAGE) Official Newsletter of the Japan Neural

         Network Society(JNNS)

         (Note: remarks by Osamu Saito "saito@nttica.NTT.JP")

Title:   Neural Networks Today

Remark:  I found this title in a bulletin board of october last year.

         It was a message of Tim Pattison, timpatt@augean.OZ

         (Note: remark provided by J.R.M. Smits "")

Title:   Computer Simulations in Brain Science

Title:   Internation Journal of Neuroscience

Title:   Neural Network Computation

Remark:  Possibly the same as "Neural Computation"

Title:   Neural Computing and Applications

Freq.:   Quarterly

Publish: Springer Verlag

Cost/yr: 120 Pounds

Remark:  Is the journal of the Neural Computing Applications Forum.

         Publishes original research and other information

         in the field of practical applications of neural computing.

B. NN Related Journals:

Title:   Complex Systems

Publish: Complex Systems Publications

Address: Complex Systems Publications, Inc., P.O. Box 6149, Champaign,

         IL 61821-8149, USA

Freq.:   6 times per year (1st volume is 1987)

ISSN #:  0891-2513

Cost/Yr: Individual $75, Institution $225

Remark:  Journal COMPLEX SYSTEMS  devotes to rapid publication of research

         on science, mathematics, and engineering of systems with simple

         components but complex overall behavior. Send mail to

         "" for additional info.

         (Remark is from announcement on Net)

Title:   Biological Cybernetics (Kybernetik)

Publish: Springer Verlag

Remark:  Monthly (vol. 1 in 1961)

Title:   Various IEEE Transactions and Magazines

Publish: IEEE

Remark:  Primarily see IEEE Trans. on System, Man and Cybernetics;

         Various Special Issues: April 1990 IEEE Control Systems

         Magazine.; May 1989 IEEE Trans. Circuits and Systems.;

         July 1988 IEEE Trans. Acoust. Speech Signal Process.

Title:   The Journal of Experimental and Theoretical Artificial Intelligence

Publish: Taylor & Francis, Ltd.

Address: London, New York, Philadelphia

Freq.:   ? (1st issue Jan 1989)

Remark:  For submission information, please contact either of the editors:

         Eric Dietrich                        Chris Fields

         PACSS - Department of Philosophy     Box 30001/3CRL

         SUNY Binghamton                      New Mexico State University

         Binghamton, NY 13901                 Las Cruces, NM 88003-0001

Title:   The Behavioral and Brain Sciences

Publish: Cambridge University Press

Remark:  (Expensive as hell, I'm sure.)

         This is a delightful journal that encourages discussion on a

         variety of controversial topics.  I have especially enjoyed

         reading some papers in there by Dana Ballard and Stephen

         Grossberg (separate papers, not collaborations) a few years

         back.  They have a really neat concept: they get a paper,

         then invite a number of noted scientists in the field to

         praise it or trash it.  They print these commentaries, and

         give the author(s) a chance to make a rebuttal or

         concurrence.  Sometimes, as I'm sure you can imagine, things

         get pretty lively.  I'm reasonably sure they are still at

         it--I think I saw them make a call for reviewers a few

         months ago.  Their reviewers are called something like

         Behavioral and Brain Associates, and I believe they have to

         be nominated by current associates, and should be fairly

         well established in the field.  That's probably more than I

         really know about it but maybe if you post it someone who

         knows more about it will correct any errors I have made.

         The main thing is that I liked the articles I read. (Note:

         remarks by Don Wunsch )

Title:   International Journal of Applied Intelligence

Publish: Kluwer Academic Publishers

Remark:  first issue in 1990(?)

Title:   Bulletin of Mathematical Biology

Title:   Intelligence

Title:   Journal of Mathematical Biology

Title:   Journal of Complex System

Title:   International Journal of Modern Physics C

Publish: USA: World Scientific Publishing Co., 1060 Main Street, River Edge,

         NJ 07666. Tel: (201) 487 9655; Europe: World Scientific Publishing

         Co. Ltd., 57 Shelton Street, London WC2H 9HE, England.

         Tel: (0171) 836 0888; Asia: World Scientific Publishing Co. Pte. Ltd.,

         1022 Hougang Avenue 1 #05-3520, Singapore 1953, Rep. of Singapore

         Tel: 382 5663.

Freq:    bi-monthly

Eds:     H. Herrmann, R. Brower, G.C. Fox and S Nose

Title:   Machine Learning

Publish: Kluwer Academic Publishers

Address: Kluwer Academic Publishers

         P.O. Box 358

         Accord Station

         Hingham, MA 02018-0358 USA

Freq.:   Monthly (8 issues per year; increasing to 12 in 1993)

Cost/Yr: Individual $140 (1992); Member of AAAI or CSCSI $88

Remark:  Description: Machine Learning is an international forum for

         research on computational approaches to learning.  The journal

         publishes articles reporting substantive research results on a

         wide range of learning methods applied to a variety of task

         domains.  The ideal paper will make a theoretical contribution

         supported by a computer implementation.

         The journal has published many key papers in learning theory,

         reinforcement learning, and decision tree methods.  Recently

         it has published a special issue on connectionist approaches

         to symbolic reasoning.  The journal regularly publishes

         issues devoted to genetic algorithms as well.

Title:   INTELLIGENCE - The Future of Computing

Published by: Intelligence

Address: INTELLIGENCE, P.O. Box 20008, New York, NY 10025-1510, USA,

212-222-1123 voice & fax; email:, CIS: 72400,1013

Freq.    Monthly plus four special reports each year (1st issue: May, 1984)

ISSN #:  1042-4296

Editor:  Edward Rosenfeld

Cost/Yr: $395 (USA), US$450 (elsewhere)

Remark:  Has absorbed several other newsletters, like Synapse/Connection

         and Critical Technology Trends (formerly AI Trends).

         Covers NN, genetic algorithms, fuzzy systems, wavelets, chaos

         and other advanced computing approaches, as well as molecular

         computing and nanotechnology.

Title:   Journal of Physics A: Mathematical and General

Publish: Inst. of Physics, Bristol

Freq:    24 issues per year.

Remark:  Statistical mechanics aspects of neural networks

         (mostly Hopfield models).

Title:   Physical Review A: Atomic, Molecular and Optical Physics

Publish: The American Physical Society (Am. Inst. of Physics)

Freq:    Monthly

Remark:  Statistical mechanics of neural networks.

Title:   Information Sciences

Publish: North Holland (Elsevier Science)

Freq.:   Monthly

ISSN:    0020-0255

Editor:  Paul P. Wang; Department of Electrical Engineering; Duke University;

         Durham, NC 27706, USA

C. Journals loosely related to NNs:


Remark:  (Must rank alongside Wolfram's Complex Systems)

Title:   IEEE ASSP Magazine

Remark:  (April 1987 had the Lippmann intro. which everyone likes to cite)


Remark:  (Vol 40, September 1989 had the survey paper by Hinton)


Remark:  (the Boltzmann machine paper by Ackley et al appeared here

         in Vol 9, 1983)


Remark:  (Vol 28, March 1988 contained the Fodor and Pylyshyn

         critique of connectionism)


Remark:  (no comment!)


Remark:  (several good book reviews)


Subject: Conferences and Workshops on Neural Networks?


Subject: Neural Network Associations?

  1. International Neural Network Society (INNS).

    INNS membership includes subscription to "Neural Networks", the official journal of the society. Membership is $55 for non-students and $45 for students per year. Address: INNS Membership, P.O. Box 491166, Ft. Washington, MD 20749.

  2. International Student Society for Neural Networks (ISSNNets).

    Membership is $5 per year. Address: ISSNNet, Inc., P.O. Box 15661, Boston, MA 02215 USA

  3. Women In Neural Network Research and technology (WINNERS).

    Address: WINNERS, c/o Judith Dayhoff, 11141 Georgia Ave., Suite 206, Wheaton, MD 20902. Phone: 301-933-9000.

  4. European Neural Network Society (ENNS)

    ENNS membership includes subscription to "Neural Networks", the official journal of the society. Membership is currently (1994) 50 UK pounds (35 UK pounds for students) per year. Address: ENNS Membership, Centre for Neural Networks, King's College London, Strand, London WC2R 2LS, United Kingdom.

  5. Japanese Neural Network Society (JNNS)

    Address: Japanese Neural Network Society; Department of Engineering, Tamagawa University; 6-1-1, Tamagawa Gakuen, Machida City, Tokyo; 194 JAPAN; Phone: +81 427 28 3457, Fax: +81 427 28 3597

  6. Association des Connexionnistes en THese (ACTH)

    (the French Student Association for Neural Networks); Membership is 100 FF per year; Activities: newsletter, conference (every year), list of members, electronic forum; Journal 'Valgo' (ISSN 1243-4825); WWW page: ; Contact:

  7. Neurosciences et Sciences de l'Ingenieur (NSI)

    Biology & Computer Science Activity : conference (every year) Address : NSI - TIRF / INPG 46 avenue Felix Viallet 38031 Grenoble Cedex FRANCE

  8. IEEE Neural Networks Council

    Web page at

  9. SNN (Foundation for Neural Networks)

    The Foundation for Neural Networks (SNN) is a university based non-profit organization that stimulates basic and applied research on neural networks in the Netherlands. Every year SNN orgines a symposium on Neural Networks. See

You can find nice lists of NN societies in the WWW at and at


Subject: On-line and machine-readable information about NNs?

    See also "Other NN links?"

  1. Neuron Digest

    Internet Mailing List. From the welcome blurb: "Neuron-Digest is a list (in digest form) dealing with all aspects of neural networks (and any type of network or neuromorphic system)" To subscribe, send email to The ftp archives (including back issues) are available from in pub/Neuron-Digest or by sending email to "". readers also find the messages in that newsgroup in the form of digests.

  2. Usenet groups (Oha!) and comp.theory.self-org-sys.

    There is a periodic posting on sent by (Gregory Aharonian) about Neural Network patents.

  3. USENET newsgroup

    Forum for discussion of academic/student-related issues in NNs, as well as information on ISSNNet (see question "associations") and its activities.

  4. Central Neural System Electronic Bulletin Board

       Supported by: Wesley R. Elsberry
                     4160 Pirates' Beach,
                     Galveston, TX 77554
       Alternative URL:
    Many MS-DOS PD and shareware simulations, source code, benchmarks, demonstration packages, information files; some Unix, Macintosh, Amiga related files. Also available are files on AI, AI Expert listings 1986-1991, fuzzy logic, genetic algorithms, artificial life, evolutionary biology, and many Project Gutenberg and Wiretap etexts.

  5. AI CD-ROM

    Network Cybernetics Corporation produces the "AI CD-ROM". It is an ISO-9660 format CD-ROM and contains a large assortment of software related to artificial intelligence, artificial life, virtual reality, and other topics. Programs for OS/2, MS-DOS, Macintosh, UNIX, and other operating systems are included. Research papers, tutorials, and other text files are included in ASCII, RTF, and other universal formats. The files have been collected from AI bulletin boards, Internet archive sites, University computer deptartments, and other government and civilian AI research organizations. Network Cybernetics Corporation intends to release annual revisions to the AI CD-ROM to keep it up to date with current developments in the field. The AI CD-ROM includes collections of files that address many specific AI/AL topics including Neural Networks (Source code and executables for many different platforms including Unix, DOS, and Macintosh. ANN development tools, example networks, sample data, tutorials. A complete collection of Neural Digest is included as well.) The AI CD-ROM may be ordered directly by check, money order, bank draft, or credit card from:
            Network Cybernetics Corporation;
            4201 Wingren Road Suite 202;
            Irving, TX 75062-2763;
            Tel 214/650-2002;
            Fax 214/650-1929;
    The cost is $129 per disc + shipping ($5/disc domestic or $10/disc foreign) (See the FAQ for further details)

  6. Machine Learning mailing list

    The Machine Learning mailing list is an unmoderated mailing list intended for people in Computer Sciences, Statistics, Mathematics, and other areas or disciplines with interests in Machine Learning. Researchers, practitioners, and users of Machine Learning in academia, industry, and government are encouraged to join the list to discuss and exchange ideas regarding any aspect of Machine Learning, e.g., various learning algorithms, data pre-processing, variable selection mechanism, instance selection, and applications to real-world problems.

    You can post, read, and reply messages on the Web. Or you can choose to receive messages as individual emails, daily summaries, daily full-text digest, or read them on the Web only.

    To subscribe, send an empty message to

    Archives are at

  7. Machine Learning Papers


Subject: How to benchmark learning methods?

The NN benchmarking resources page at was created after a NIPS 1995 workshop on NN benchmarking. The page contains pointers to various papers on proper benchmarking methodology and to various sources of datasets.

Benchmark studies require some familiarity with the statistical design and analysis of experiments. There are many textbooks on this subject, of which Cohen (1995) will probably be of particular interest to researchers in neural nets and machine learning (see also the review of Cohen's book by Ron Kohavi in the International Journal of Neural Systems, which can be found on-line at


Cohen, P.R. (1995), Empirical Methods for Artificial Intelligence, Cambridge, MA: The MIT Press.


Subject: Databases for experimentation with NNs?

  1. UCI machine learning database

    A large collection of data sets accessible via anonymous FTP at [] in directory /pub/machine-learning-databases" or via web browser at

  2. UCI KDD Archive

    The UC Irvine Knowledge Discovery in Databases (KDD) Archive at is an online repository of large datasets which encompasses a wide variety of data types, analysis tasks, and application areas. The primary role of this repository is to serve as a benchmark testbed to enable researchers in knowledge discovery and data mining to scale existing and future data analysis algorithms to very large and complex data sets. This archive is supported by the Information and Data Management Program at the National Science Foundation, and is intended to expand the current UCI Machine Learning Database Repository to datasets that are orders of magnitude larger and more complex.

  3. The neural-bench Benchmark collection

    Accessible WWW at or via anonymous FTP at In case of problems or if you want to donate data, email contact is "". The data sets in this repository include the 'nettalk' data, 'two spirals', protein structure prediction, vowel recognition, sonar signal classification, and a few others.

  4. Proben1

    Proben1 is a collection of 12 learning problems consisting of real data. The datafiles all share a single simple common format. Along with the data comes a technical report describing a set of rules and conventions for performing and reporting benchmark tests and their results. Accessible via anonymous FTP on [] as /afs/cs/project/connect/bench/contrib/prechelt/proben1.tar.gz. and also on as /pub/neuron/proben1.tar.gz. The file is about 1.8 MB and unpacks into about 20 MB.

  5. Delve: Data for Evaluating Learning in Valid Experiments

    Delve is a standardised, copyrighted environment designed to evaluate the performance of learning methods. Delve makes it possible for users to compare their learning methods with other methods on many datasets. The Delve learning methods and evaluation procedures are well documented, such that meaningful comparisons can be made. The data collection includes not only isolated data sets, but "families" of data sets in which properties of the data, such as number of inputs and degree of nonlinearity or noise, are systematically varied. The Delve web page is at

  6. NIST special databases of the National Institute Of Standards And Technology:

    Several large databases, each delivered on a CD-ROM. Here is a quick list. Here are example descriptions of two of these databases:

    NIST special database 2: Structured Forms Reference Set (SFRS)

    The NIST database of structured forms contains 5,590 full page images of simulated tax forms completed using machine print. THERE IS NO REAL TAX DATA IN THIS DATABASE. The structured forms used in this database are 12 different forms from the 1988, IRS 1040 Package X. These include Forms 1040, 2106, 2441, 4562, and 6251 together with Schedules A, B, C, D, E, F and SE. Eight of these forms contain two pages or form faces making a total of 20 form faces represented in the database. Each image is stored in bi-level black and white raster format. The images in this database appear to be real forms prepared by individuals but the images have been automatically derived and synthesized using a computer and contain no "real" tax data. The entry field values on the forms have been automatically generated by a computer in order to make the data available without the danger of distributing privileged tax information. In addition to the images the database includes 5,590 answer files, one for each image. Each answer file contains an ASCII representation of the data found in the entry fields on the corresponding image. Image format documentation and example software are also provided. The uncompressed database totals approximately 5.9 gigabytes of data.

    NIST special database 3: Binary Images of Handwritten Segmented Characters (HWSC)

    Contains 313,389 isolated character images segmented from the 2,100 full-page images distributed with "NIST Special Database 1". 223,125 digits, 44,951 upper-case, and 45,313 lower-case character images. Each character image has been centered in a separate 128 by 128 pixel region, error rate of the segmentation and assigned classification is less than 0.1%. The uncompressed database totals approximately 2.75 gigabytes of image data and includes image format documentation and example software.

    The system requirements for all databases are a 5.25" CD-ROM drive with software to read ISO-9660 format. Contact: Darrin L. Dimmick;; (301)975-4147

    The prices of the databases are between US$ 250 and 1895 If you wish to order a database, please contact: Standard Reference Data; National Institute of Standards and Technology; 221/A323; Gaithersburg, MD 20899; Phone: (301)975-2208; FAX: (301)926-0416

    Samples of the data can be found by ftp on in directory /pub/data A more complete description of the available databases can be obtained from the same host as /pub/databases/catalog.txt

  7. CEDAR CD-ROM 1: Database of Handwritten Cities, States, ZIP Codes, Digits, and Alphabetic Characters

    The Center Of Excellence for Document Analysis and Recognition (CEDAR) State University of New York at Buffalo announces the availability of CEDAR CDROM 1: USPS Office of Advanced Technology The database contains handwritten words and ZIP Codes in high resolution grayscale (300 ppi 8-bit) as well as binary handwritten digits and alphabetic characters (300 ppi 1-bit). This database is intended to encourage research in off-line handwriting recognition by providing access to handwriting samples digitized from envelopes in a working post office.
         Specifications of the database include:
         +    300 ppi 8-bit grayscale handwritten words (cities,
              states, ZIP Codes)
              o    5632 city words
              o    4938 state words
              o    9454 ZIP Codes
         +    300 ppi binary handwritten characters and digits:
              o    27,837 mixed alphas  and  numerics  segmented
                   from address blocks
              o    21,179 digits segmented from ZIP Codes
         +    every image supplied with  a  manually  determined
              truth value
         +    extracted from live mail in a  working  U.S.  Post
         +    word images in the test  set  supplied  with  dic-
              tionaries  of  postal  words that simulate partial
              recognition of the corresponding ZIP Code.
         +    digit images included in test  set  that  simulate
              automatic ZIP Code segmentation.  Results on these
              data can be projected to overall ZIP Code recogni-
              tion performance.
         +    image format documentation and software included
    System requirements are a 5.25" CD-ROM drive with software to read ISO-9660 format. For further information, see or send email to Ajay Shekhawat at <ajay@cedar.Buffalo.EDU>

    There is also a CEDAR CDROM-2, a database of machine-printed Japanese character images.

  8. AI-CD-ROM (see question "Other sources of information")

  9. Time series archive

    Various datasets of time series (to be used for prediction learning problems) are available for anonymous ftp from [] in /pub/Time-Series". Problems are for example: fluctuations in a far-infrared laser; Physiological data of patients with sleep apnea; High frequency currency exchange rate data; Intensity of a white dwarf star; J.S. Bachs final (unfinished) fugue from "Die Kunst der Fuge"

    Some of the datasets were used in a prediction contest and are described in detail in the book "Time series prediction: Forecasting the future and understanding the past", edited by Weigend/Gershenfield, Proceedings Volume XV in the Santa Fe Institute Studies in the Sciences of Complexity series of Addison Wesley (1994).

  10. USENIX Faces

    The USENIX faces archive is a public database, accessible by ftp, that can be of use to people working in the fields of human face recognition, classification and the like. It currently contains 5592 different faces (taken at USENIX conferences) and is updated twice each year. The images are mostly 96x128 greyscale frontal images and are stored in ascii files in a way that makes it easy to convert them to any usual graphic format (GIF, PCX, PBM etc.). Source code for viewers, filters, etc. is provided. Each image file takes approximately 25K.

    For further information, see Do NOT do a directory listing in the top directory of the face archive, as it contains over 2500 entries!

    According to the archive administrator, Barbara L. Dijker (, there is no restriction to use them. However, the image files are stored in separate directories corresponding to the Internet site to which the person represented in the image belongs, with each directory containing a small number of images (two in the average). This makes it difficult to retrieve by ftp even a small part of the database, as you have to get each one individually.
    A solution, as Barbara proposed me, would be to compress the whole set of images (in separate files of, say, 100 images) and maintain them as a specific archive for research on face processing, similar to the ones that already exist for fingerprints and others. The whole compressed database would take some 30 megabytes of disk space. I encourage anyone willing to host this database in his/her site, available for anonymous ftp, to contact her for details (unfortunately I don't have the resources to set up such a site).

    Please consider that UUNET has graciously provided the ftp server for the FaceSaver archive and may discontinue that service if it becomes a burden. This means that people should not download more than maybe 10 faces at a time from uunet.

    A last remark: each file represents a different person (except for isolated cases). This makes the database quite unsuitable for training neural networks, since for proper generalisation several instances of the same subject are required. However, it is still useful for use as testing set on a trained network.

  11. Linguistic Data Consortium

    The Linguistic Data Consortium (URL: is an open consortium of universities, companies and government research laboratories. It creates, collects and distributes speech and text databases, lexicons, and other resources for research and development purposes. The University of Pennsylvania is the LDC's host institution. The LDC catalog includes pronunciation lexicons, varied lexicons, broadcast speech, microphone speech, mobile-radio speech, telephone speech, broadcast text, conversation text, newswire text, parallel text, and varied text, at widely varying fees.

    Linguistic Data Consortium 
    University of Pennsylvania 
    3615 Market Street, Suite 200 
    Philadelphia, PA 19104-2608 
    Tel (215) 898-0464 Fax (215) 573-2175

  12. Otago Speech Corpus

    The Otago Speech Corpus contains speech samples in RIFF WAVE format that can be downloaded from

  13. Astronomical Time Series

    Prepared by Paul L. Hertz (Naval Research Laboratory) & Eric D. Feigelson (Pennsyvania State University): URL:

  14. Miscellaneous Images

    The USC-SIPI Image Database:

    CityU Image Processing Lab:

    Center for Image Processing Research:

    Computer Vision Test Images:

    Lenna 97: A Complete Story of Lenna:

  15. StatLib

    The StatLib repository at at Carnegie Mellon University has a large collection of data sets, many of which can be used with NNs.


Next part is part 5 (of 7). Previous part is part 3.